What is S3 storage and S3 compatible storage providers

In 2019, as the founder of gaathastory, amy venture as a audio storytelling platform, I was on a mission to scale our digital infrastructure without draining our startup budget. The breakthrough came through the a startup program, which gifted us AWS credits—like finding a discount voucher for a beloved eatery.

These credits unlocked Amazon S3, a cloud storage service that promised to safeguard our growing library of audiobooks and images. My first encounter with the AWS console was a whirlwind of excitement and confusion, and a steep learning curve.

My experience with S3 compatible storage providers

This post chronicles my personal journey with cloud storage – from first encountering AWS S3 through startup credits, to exploring cost-effective s3 compatible storage alternatives like Backblaze B2, integrating with Cloudflare, and finally setting up efficient workflows using rclone. I share practical lessons, humorous moments, and technical insights gained over five years of hands-on experience.

हिंदी सारांश

इस ब्लॉग में मैं अपनी क्लाउड स्टोरेज यात्रा के बारे में बताता हूँ – कैसे नैसकॉम स्टार्टअप प्रोग्राम से मिले AWS क्रेडिट्स से शुरुआत हुई, फिर बैकब्लेज़ B2 जैसे किफायती विकल्पों की खोज, क्लाउडफ्लेयर के साथ एकीकरण, और अंत में rclone का उपयोग करके कुशल वर्कफ़्लो सेटअप करना। मैंने पांच साल के अनुभव से प्राप्त व्यावहारिक सबक, मज़ेदार पल और तकनीकी जानकारियाँ साझा की हैं।

मराठी सारांश

या ब्लॉगमध्ये मी माझा क्लाउड स्टोरेज प्रवास सांगतो – कसा नॅसकॉम स्टार्टअप प्रोग्रामद्वारे मिळालेल्या AWS क्रेडिट्सपासून सुरुवात झाली, नंतर बॅकब्लेझ B2 सारखे किफायतशीर पर्याय शोधणे, क्लाउडफ्लेअरसह एकत्रीकरण, आणि शेवटी rclone वापरून कार्यक्षम वर्कफ्लो सेटअप करणे. मी पाच वर्षांच्या अनुभवातून मिळवलेले व्यावहारिक धडे, विनोदी क्षण आणि तांत्रिक अंतर्दृष्टी शेअर केली आहे.

Introduction: How My Cloud Storage Adventure Began

My journey into the world of S3 and cloud storage started quite unexpectedly back in 2019. As a bootstrapped founder of GaathaStory, I was perpetually hunting for ways to manage expenses while scaling our digital infrastructure. Fortune smiled when we got selected for the startup program, which came with a generous helping of AWS credits. Like finding a surprise discount coupon for your favorite restaurant, these credits opened doors to AWS services that would have otherwise burnt a hole in our modest startup wallet.

I still remember staring at the AWS console for the first time – a confused mix of excitement and overwhelm, much like a child who has been handed the keys to a toy store but doesn’t know where to begin. S3 (Simple Storage Service) caught my attention immediately as we needed reliable storage for our growing collection of audiobooks and images.

My journey into the world of S3 and cloud storage started quite unexpectedly back in 2019. As a bootstrapped founder of gaathaStory, I was perpetually hunting for ways to manage expenses while scaling our digital infrastructure. Fortune smiled when we got selected for the startup program, which came with a generous helping of AWS credits. Like finding a surprise discount coupon for your favorite restaurant, these credits opened doors to AWS services that would have otherwise burnt a hole in our modest startup wallet.

I still remember staring at the AWS console for the first time – a confused mix of excitement and overwhelm, much like a child who has been handed the keys to a toy store but doesn’t know where to begin. S3 (Simple Storage Service) caught my attention immediately as we needed reliable storage for our growing collection of audiobooks and images.

Learning the Ropes: YouTube University

With AWS credits in hand and zero expertise, I turned to where most self-taught tech enthusiasts go – YouTube. Back in 2018-2019, two tutorials became my digital gurus:

1. AWS S3 Basics for Beginners by Be A Better Dev
2. S3 Deep Dive with Examples

These videos were like finding an English-speaking guide in a foreign country – suddenly everything started making sense! I spent countless evenings after office hours setting up buckets, configuring permissions, and uploading files.

The honeymoon period with AWS was beautiful while the credits lasted. Then came the bills. Anyone who’s transitioned from free credits to paying AWS bills knows that feeling of shock. It reminded me of Erlich Bachman’s memorable rant in Silicon Valley about AWS fees – “I’m not paying because you’re not the one getting [bleeped] face first by your credit card company because of massive AWS hosting fees!”

While not quite as dramatic as Erlich’s experience, our modest startup definitely felt the pinch when the credits dried up. As a bootstrapped company with Indian pricing for our services, AWS costs were disproportionately high compared to our revenue.

The Search for Alternatives: Enter Backblaze B2

The search for alternatives began like most frugal Indian entrepreneurs’ quests – with Google searches for “AWS S3 alternatives cheaper” and “S3 compatible storage India pricing.” This exploration led me to Backblaze B2 – a service offering S3-compatible storage at nearly one-fourth the cost of AWS.

The migration wasn’t without hiccups. Documentation was sparse compared to AWS, and there were moments of doubt. Would a lesser-known service be reliable enough for production use? What if something went wrong? The internal debate resembled deciding between a trusted but expensive Swiggy order versus trying the new local delivery app with steep discounts.

Eventually, cost considerations won, and we took a calculated risk with B2. The initial setup was remarkably straightforward:

# Example of creating a B2 bucket and uploading files
b2 create-bucket my-gaathastory-bucket allPrivate
b2 upload-file my-gaathastory-bucket ./local-audio-file.mp3 uploaded-audio.mp3

Gumlet Integration: The Game Changer

One of our biggest challenges was efficiently serving user-uploaded images across different device resolutions. This is where Gumlet came into our technical stack. Rather than storing multiple versions of each image, we configured Gumlet to fetch original images from our B2 bucket, process them on-the-fly, and serve optimized versions.

The integration was surprisingly smooth:

1. Created a dedicated B2 bucket for user uploads
2. Connected Gumlet to this bucket as the source
3. Updated our frontend code to request images through Gumlet URLs
4. Watched our page load times drop significantly!

This single integration solved multiple problems – image optimization, responsive sizing, and fast delivery. The best part? The cost was a fraction of what we would have paid with an AWS-only solution.

Cloudflare Bandwidth Alliance: The Cost-Saving Discovery

About a year into our B2 journey, I stumbled upon information about the Cloudflare prompted Bandwidth Alliance while scrolling through Hacker News during a particularly boring conference call. The proposition seemed almost too good to be true: zero data transfer fees when serving content from B2 through Cloudflare.

For anyone unfamiliar with cloud pricing, egress fees (charges for data leaving the cloud provider) can often exceed storage costs, especially for media-heavy applications like ours. This alliance meant we could potentially save thousands of rupees monthly.

Setting up the integration required some DNS gymnastics:

1. Created a CNAME record in Cloudflare pointing to our B2 bucket
2. Configured Cloudflare to proxy the traffic
3. Updated our application to use the new Cloudflare URLs

# Example Cloudflare DNS Record
media CNAME f002.backblazeb2.com

The results were immediate and impressive. Our monthly bill dropped significantly, and the content delivery actually became faster thanks to Cloudflare’s extensive global network.

Exploring Cloudflare R2: The New Contender

In early 2023, Cloudflare introduced their own S3 compatible storage called R2. The proposition was enticing – storage pricing comparable to B2 but with zero egress fees built-in, without requiring additional configuration. As someone who’s always keen to optimize our tech stack, I had to explore this option.

R2 is particularly attractive if you’re already using other Cloudflare services. The integration is seamless, and having everything under one dashboard reduces cognitive overhead – no more jumping between multiple services to debug issues.

My testing revealed that R2 performed admirably for our use cases, though migration from an established B2 setup would require some effort. It’s definitely worth considering for new projects or if you’re already heavily invested in the Cloudflare ecosystem.

Rclone: The Swiss Army Knife for Cloud Storage

Managing data across different storage providers eventually led me to rclone – an open-source command-line tool that’s like rsync but for cloud storage. It has become an essential part of our toolkit for:

1. Backup automation
2. Migrating data between providers
3. Synchronizing content across environments

Setting up rclone for B2 is straightforward:

# Install rclone
$ curl https://rclone.org/install.sh | sudo bash

# Configure B2 remote
$ rclone config
# Follow interactive prompts to add B2 credentials

# Sync local folder to B2
$ rclone sync /local/path remote:bucket/path --progress

The beauty of rclone is its consistency across providers. Once you learn the basic commands, they work similarly whether you’re using B2, R2, S3, or any of the dozens of supported services.

The Bandwidth Alliance, spearheaded by Cloudflare, is a game-changer for startups and businesses looking to cut data transfer costs without compromising performance. By partnering with cloud storage providers like Backblaze and Scaleway, the Alliance makes moving data between services free or significantly cheaper.

When I first tested Backblaze be interconnect with cloudflare for my audio storytelling platform,gaathastory, I was amazed at the ease of setup.

What is the Bandwidth Alliance?

Imagine the Bandwidth Alliance as a cooperative of cloud companies working together like a well-organized Indian market, where vendors share resources to keep costs low for customers. Led by Cloudflare, it includes providers like Backblaze B2 and Scaleway, ensuring data transfers between their services and Cloudflare’s network are free or heavily discounted. This means you can store files in a “digital dabba” (a bucket) and serve them to your website visitors without hefty bandwidth fees. It’s a win-win for startups and bloggers watching their budgets.

Key Takeaways from My Journey

After five years of navigating the cloud storage landscape, here are my key insights:

1. Start small but plan for scale – Begin with what you need today, but choose solutions that can grow with you
2. Cost optimization requires vigilance – Regularly review your usage patterns and adjust accordingly
3. Integration capabilities matter – How well a service works with your existing stack is often more important than features in isolation
4. Build for resilience – Design your systems assuming any single provider might fail
5. Keep learning – The cloud landscape evolves rapidly; what’s optimal today may not be tomorrow

Where I Stand Today

My s3 compatible storage journey has evolved from wide-eyed wonder at AWS services to a pragmatic multi-provider approach optimized for our specific needs. Today, we use a combination of:

• Gumlet and PushrCDN for image processing and optimization
• Rclone for automation and management

This hybrid approach gives us the best balance of cost, performance, and reliability. The most valuable lesson I’ve learned is that there’s rarely a one-size-fits-all solution in technology – the best architecture is one tailored to your specific requirements, constraints, and growth trajectory.

FAQ: Quick Answers Before We Dive Deep

Q1: Is AWS S3 the most cost-effective storage solution for startups?
A1: Not necessarily. While AWS S3 offers reliability and scalability, alternatives like Backblaze B2 can be significantly more affordable, especially for storage-heavy applications.

Q2: Do I need technical expertise to implement s3 compatible storage solutions?
A2: Basic technical understanding helps, but many solutions now have user-friendly interfaces. This post shares my self-taught journey using freely available tutorials and documentation.

Q3: What’s the biggest advantage of using Cloudflare with B2 storage?
A3: The Cloudflare Bandwidth Alliance eliminates egress fees when serving content from B2 through Cloudflare, potentially saving thousands of rupees monthly for high-traffic sites.

Related Posts

• Set Up Image and Video CDN Using Gumlet: Learn how to optimize content delivery with Backblaze B2.
• How to Use Rclone for Backups and Cloud Storage: Streamline data transfers with rclone.
• How to Select the Right Web Hosting for Your Blog: Choose hosting that pairs with the Bandwidth Alliance.
• Your Guide to Amazon S3: Explore S3 and its alternatives for storage.

Links and Resources for S3 Comaptible Storage

The cloud storage landscape continues to evolve, with new providers and services emerging regularly. What works best for your situation will depend on your specific requirements, budget constraints, and technical capabilities. I hope sharing my journey helps you navigate your own path through the sometimes confusing world of cloud storage options.