cgalea11 Member Posts: 13

Hi All,

Has anyone used PyArrow on Domino. I'd like to speed up the conversion to pandas dataframes in PySpark using the .toPandas() command.

Thank you,



  • dan.stern
    dan.stern Member, Moderator, Domino Posts: 37 mod

    Hi Charles,

    It looks like pyarrow should be installable in your workspace with just a

    pip install pyarrow

    Have you tried this and run into some difficulties with Pyarrow?

  • cgalea11
    cgalea11 Member Posts: 13

    Hi Dan,

    All good thanks. It seems to be running fine (please see code fragment below) showing a substantial reduction in time taken to convert from a Spark dataframe to pandas dataframe.


Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!