Buckaroo continues adding features including:.Histograms.Polars Support .Colab/VSCode support .Pluggable Analysis Framework (customize summary stats) .Auto-cleaning .Many other refinements. pip install buckaroo.

All of this will be part of my article "so you want to serialize a dataframe to JS". Which is a subsection (probably the largest) of "so you want to write a table viewer".

This should be much more modular.

The final cool idea though is to leverage multi indexes and polars structs. It would work like this. Say you want to color a column based on the diff with an original column. Put them into a struct, then render, coloring based on the struct column you don't display.

I am currently kicking around two ideas. no, maybe 3, writing helps. First would be a conversion step in python, plumbing this in is a mess. Second is different renderers, that do the same thing in the frontend. This is actually a bit cleaner.

But how to deal with custom coded apps that build color_maps via column comparison. By the time this gets to the frontend, those column names won't exist in the data.

The other key change is converting every column name from the original to a letter based encoding sequential encoding (a-z, aa, ab,). and then having each config for columns explicitly state header_name and field.

Rather than specialcasing pandas parquet. I am doing the following. Explicit `first_col_config` for table configuration. no more including "index" in summary stats (I don't currently have a way to display it either.

This works until you want a column named "index", or you serialize a dataframe to parquet with multi-index columns. Parquet changes "index" to the string "('index', '')".

for JSON, buckaroo serializes dataframes as a list of dicts, `to_json(orient='records')` or `polars.with_row_index()`. This always adds a column of "index" to the records. Parquet behaves similarly, but column oriented.

Hit a snag with the multi-index refactoring. The short of it is that dataframe serialization is really tricky and relying on assumption and default behavors will bite you, always.

Got to 600 today. I hadn't realized it took that long to get to 100. Still a vanity metric.

@union_ai Play with it live on marimo here

I also built an integration between #pandera and Buckaroo so that you can see validation errors from a Pandera schema in notebooks. @union_ai @cosmicbboy

All of this reminds me to finish and publish my article "So you want to write a dataframe table". It details most of the little pitfalls that I remember in building Buckaroo. it's 2k words right now.

Finally I remembered while looking this up that pandas can have multi-indexes for rows too. I can do that too, but down the line. The next major bit of assumption jank to pull out of Buckaroo after multi-indexes is pandas indexes. Lot's of implicit stuff around it.

Also, this should work really well with polars tuple types.

I needed to use multi indexes to deal with extracting data from panderas validations last week. The UX of passing a dataframe to buckaroo and seeing only a stacktrace was really bad. A little reading of AG-Grid docs and here I am.

I'm kneedeep in adding proper MultiIndex column support to Buckaroo. MIs are kind of a forgotten corner case of pandas, super powerful. Many tables don't display them properly because they are tricky. They don't serialize to JSON natively. #python #pandas #datascience #pydata

I take it back, this might be more of an alumni donation pitch. That would make more sense, because the development guy doesn't know tech/VC well.

I'm in a nieghborhood hotel getting coffee. Listening to an inhouse college VC pitch a position to an experienced VC who's an alumni. I hear business meetings at this hotel regularly. The notable ones go poorly, you can tell when the sales guy talks too much.