PyArrow

cgalea11
cgalea11 Member Posts: 13

Hi All,

Has anyone used PyArrow on Domino. I'd like to speed up the conversion to pandas dataframes in PySpark using the .toPandas() command.

Thank you,

Charles

Answers

  • dan.stern
    dan.stern Member, Moderator, Domino Posts: 37 mod

    Hi Charles,

    It looks like pyarrow should be installable in your workspace with just a

    pip install pyarrow
    

    Have you tried this and run into some difficulties with Pyarrow?

  • cgalea11
    cgalea11 Member Posts: 13

    Hi Dan,

    All good thanks. It seems to be running fine (please see code fragment below) showing a substantial reduction in time taken to convert from a Spark dataframe to pandas dataframe.

    Thanks


Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!