Arvind
@arvtalkscloud
Followers
466
Following
160
Media
34
Statuses
159
@ucberkeley Alum ✦ 3x AWS Certified & Enthusiast ✦ Building Autolake — an Autonomous Data Lake platform
San Francisco, CA
Joined November 2024
Just finalized job-level/overview cost breakdowns for Autolake. Now you can see exactly which ingestion jobs are burning your AWS credits LOL
2
0
20
When I finally think I got infra costs under control. AWS: “You’ve exceeded 85% of your usage”
0
0
7
I passed the AWS Cloud Practitioner Cert in 7 days. Here's exactly how you can too:.• day 1: Understand AWS core services (VPC, S3, EC2, Lambda).• day 2: Learn IAM, Pricing models, Global Infrastructure.• day 3: Deeper dive into EC2, S3, and Lambda.• day 4: Learn AWS
2
0
8
What was your biggest surprise cost on AWS, and how did you remove it?.
2
0
5
Schema evolution is inevitable. The trick isn’t avoiding it, it’s designing for it.
0
0
8
4/ At Autolake, our engine automates this entire process to determine the optimal balance of time & workers between DPU & cost so that your business can simply onboard any volume of data for a fraction of the typical cost. Say goodbye to manual tuning and overnight jobs.
0
0
2
3/ After the partitions have stamped the records as insert/update/deletes for CDC purposes, I utilized a Lazy Initialization Design Pattern to perform transformations to the records. Less wasted compute, faster commits.
1
0
2
2/ I had to find a balance between cost and processing time where more processing power doesn't result in a higher cost, and less processing power doesn't result in more processing time. Then used hash keys, to optimize record type comparisons. .
1
0
2
1/ First, I optimized the number of partitions in my Spark dataframe to correctly determine the # of workers allocated for the pipeline. Too many = tiny files; too few = idle workers.
1
0
2
How I ingested ~540,000,000+ records in 1 hr and 22 min for my Client's Data Lake. Costing them less than a $100. Here's how (thread):
1
1
12
me when I realize halfway during the meeting that it could have been an email
1
0
5
Got my AWS Solutions Architect cert @ 18 Years Old. But here’s what actually mattered:. Not the cert, but the confidence to build on AWS when I originally didn’t even know wtf a Lambda function was. Don’t study to pass. Build until passing is a side effect.
1
0
12
My friend was asked by a job recruiter if he had 15 years or more of AWS Glue experience. Glue launched in 2017. .
3
0
9
AI won’t take your job, but the dev who knows how to use it better than you will.
1
0
5
data lake ≠ dumping grounds. A real lake:.- has structure.- has schema validation.- doesn’t break when a CSV has a rogue semicolon. start treating your lake like production infra, not a trash bin.
1
0
2
honestly the most underrated coding skill is writing code your future self won’t hate.
2
0
5
We got GPT5 before GTA6.
GPT-5 is the smartest model we've ever done, but the main thing we pushed for is real-world utility and mass accessibility/affordability. we can release much, much smarter models, and we will, but this is something a billion+ people will benefit from. (most of the world has.
1
0
3
AI model releases have become our generation’s version of how iPhone launches pre-2018 felt like.
1
0
2