Has anyone used PyArrow on Domino. I'd like to speed up the conversion to pandas dataframes in PySpark using the .toPandas() command.
It looks like pyarrow should be installable in your workspace with just a
pip install pyarrow
Have you tried this and run into some difficulties with Pyarrow?
All good thanks. It seems to be running fine (please see code fragment below) showing a substantial reduction in time taken to convert from a Spark dataframe to pandas dataframe.
It looks like you're new here. If you want to get involved, click one of these buttons!