AWS S3 Glacier Deep Archive

I've been meaning to use AWS Glacier Deep Archive for years because it's so inexpensive. I just uploaded 780.7 GB of data into S3 using Glacier Deep Archive, which will cost me 77¢/month compared to $17.96 if I had stored this data as typical S3 Standard storage. 

The tradeoff is that retrieval of my data from Glacier Deep Archive takes either 9 – 12 hours or up to 48 hours depending how much I'm willing to spend to retrieve this data ($5.07 vs $1.56, respectively). 

This type of storage is ideal for backing up large amounts of data. In my case, I backed up my entire Apple Photos library consisting of 176,470 Photos and 6,184 Videos.

Challenges

Copying hundreds of gigabytes over the Internet is not a trivial matter. Not only does it require a stable Internet connection for a dozen hours, but one must also consider if your ISP has bandwidth limits, such as Cox, with a 1,024 GB monthly limit. (ISPs that use fiber optics generally don't have a monthly bandwidth limit.)

Initially, I copied my Photos Library.photoslibrary to my iCloud drive in hopes that it would be synced/backed up to iCloud. But, macOS exempts this file from being synced to iCloud since the iCloud Photos service is specifically designed to backup photos and share them across multiple devices.

A simple alternative solution, in theory, would be to compress (ZIP) my Photos Library.photoslibrary "file" (it's technically not a single file, but many files packaged up to look like a single file in Finder). This would have been my first option except my computer's hard drive doesn't have an extra 800GB to store the zipped up file prior to uploading it. 

The best option I found, without the need to install any other software, was to use the Unix tar command while pipping the output directly to my AWS bucket. 

tar -cf - "./Photos Library.photoslibrary" | \

aws s3 cp - \

s3://joe-apple-photos/Photos_Library_Backup.tar \

--storage-class DEEP_ARCHIVE

It took me two attempts to pipe the data to S3. The first attempt failed with the following error:

An error occurred (InvalidArgument) when calling the UploadPart operation: Part number must be an integer between 1 and 10000, inclusive

I fixed this issue by running the following AWS CLI command to change the multipart chunk size:

aws configure set \

default.s3.multipart_chunksize 512MB

Bonus Points

One downside to this technique is that I could only guesstimate how long the upload would take since there's no progress indicator. Next time I do this, I'll install and use the pv (pipe viewer) command:

tar -cf - "./Photos Library.photoslibrary" \
| pv -s 700g | \
aws s3 cp - \
s3://joe-apple-photos/Photos_Library_Backup.tar\
 --storage-class DEEP_ARCHIVE

I haven't used PV before, but the progress indicator will look like this:

142GiB 0:03:15 [12.8MiB/s] 

[==============>                             ] 20% ETA 12:08:45

OJ and & Knicks

A close friend texted this me:

The most recent time when the Knicks led in the NBA Finals was June 17, 1994 (they would lose the series), and this occurred on that same day. That was exactly 2 weeks before my I-Day. (Induction Day is when he was sworn into active duty at the start of his Plebe year at the U.S. Naval Academy).

I, myself, clearly remember that day. I had just finished MOS school and was driving from New York to Southern California to report to my permanent duty station at Camp Pendleton.

When I left New York on Monday, I made it to Kent State and saw O.J. in handcuffs on the evening news.

That Friday, when I arrived in Los Angeles to stay with a high school buddy for the weekend, there was a BOLO for O.J. My arrival in L.A. coincided with the manhunt for O.J. and the ensuing white Ford Bronco chase.

Since my buddy and I were from New York, we were trying to watch the Knicks game, but the network put the game in a small box on the screen. We couldn’t see the score because the main coverage was focused on the 35-mph car chase.

Foreign Intelligence Gathering Through Facebook

There’s an old joke that the CIA set up Facebook for human intelligence gathering since it’s far easier to have people willingly share the details of their lives rather than expend the energy themselves to collect it through surveillance.

Obviously, the CIA has nothing to do with the origins of Facebook, but foreign governments do use it for gathering intelligence, especially through government and military affiliated Facebook groups. 

Here’s a Facebook group purporting to be an official U.S. Marine Corps group which has over 19,000 members.  

On the surface, this group seems legitimate and many former Marines, like myself, have joined it. But I became suspicious when I saw posts like the following. (An MOS is a military member’s job that they were formally trained for.)  

These two names are South Asian Muslim and Indonesian, respectively, which is far from the typical Marine Corps demographic. Sakib has locked his own Facebook profile, a sign that is a red flag indicating a scammer. Kristin’s Facebook profile is a advertising agency that was created last month with five posts and eight followers with no connection to the Marine Corps. 

Let’s take a look at this Facebook group’s admin: 

Rubel Miah, the group’s admin, is a long way from having any connection with an official U.S. Marine Corps page.

Creating a Facebook group with nearly 20,000 members takes work; and it doesn’t take long to realize that the creators of this group have nothing to do with the Marine Corps. So why go through all this trouble? Someone’s harvesting intel and AI certainly makes it much easier to collect.

Intel collection through social media has been around since day one. I’m surprised how little effort this Facebook group put into this endeavor, but thanks to mindless doom scrolling, it’s working. 

I reached out to Meta (Facebook’s parent company) for comment but haven’t heard back. 

Filtering News

Most national and international news isn't actionable by us as individuals, so do we really need to watch it? 

In 2018, I wrote about how to deal with the effects of the continuous flow of bad news.

I have three rules for filtering news:

1. How will it affect my lifestyle?

2. How will it affect my daily activities? 

3. How will it change my mind in the voting booth?

For you and friends/family, if the answer to all three is, "Not much," then ask yourself why you're continually watching the news? Bored? Looking for excitement? Curious?

Keep in mind what it's doing to your mental health.

 

Many people end up watching the news and then complaining on social media about it which spreads the angst. If you really want to do something, then do something actionable. Instead of going to a protest, contact your member of Congress. Do you even know who your three members of Congress are? Mine are Padilla and Schiff in the Senate and Peters in the House. 

Better yet, watch the local news. Everything actionable will be covered there and your local government officials are easier to access when you want to be heard. 

Having simple rules helps. Here are My Three Daily Life Goals.

AI Agents For Hire

In a very near future, corporations will emerge with cultivated AI agents that a person or company could buy, license, or subscribe to in order to do specific tasks. Instead of hiring employees, especially at the junior level, a company could instead use HR, payroll, or organizational AI agents to do that work while putting a human in the loop to supervise and review the final work product.

AI is coming close to knowing all world knowledge similar to humanity in the TV show, Pluribus. 

Soon, we may see professionals who’ve trained their own AI agents as extensions of themselves. These individuals could be hired alongside their agents, or even operate as solo CEOs with no traditional staff, deploying AI “employees” to execute work for other companies. In some cases, hiring organizations might interview the AI agent themselves, much like they would a human candidate, before hiring someone.

AI Agents Run Amok: Clawdbot –> Moltbot –> OpenClaw

It looks like we've settled on a name...

What was called Clawdbot, and renamed to Moltbot, is finally called OpenClaw; and it's dangerous if you don't know what you're doing. 

It's doxed people and scammed them out of money very quickly. This video explains the details that moved markets, today, such as Apple and Cloudflare stock. 

Claude for Coding a Website

I was listening to last Friday's Hard Fork tech podcast where the two hosts were discussing how they're saving $89/year by no longer hosting their personal websites on Squarespace; instead they using Claude Code.

Today, I tried Claude Code out myself to modernize my personal website, joemoreno.com, which I originally made with iWeb nearly 20 years ago. Its design, look, and feel were long in the tooth. iWeb, which was discontinued about five years later, was Apple's answer to Microsoft FrontPage. These two tools allowed consumers to create static webpages with the ease of creating a Keynote or PowerPoint presentation. 

To make this happen, I simply used Claude from a web browser along with this prompt:

Take the content in these files at http://joemoreno.com/index.html and recreate the same website but with a modern look and feel.

Claude then coded for less than five minutes, and boom, it was done. I could render the website both inside the Claude environment and also publish it with a public URL hosted on Claude. 

I then downloaded the index.html, which had the HTML and CSS in a single file, uploaded it to my AWS S3 bucket and I was done.

Now that was quick and easy. 

AI Meets Expectations: Personal Turing Test

Left: What I got. Right:What it should look like

I’ve had two personal tests that I’ve applied to every new AI model, and for years none of them passed.

One test was to generate an image of an absinthe fountain with glasses and absinthe spoons. Last year, AI got this right. 

The other test was to take a multi-hours MP3 that I recorded at monthly jazz jam sessions and break it into individual tracks based on either silence between the songs or applause. This week, ChatGPT finally got it right, but I have yet to see success with other AI models. 

Coding Tips: Stop Vibe Coding – Start Writing Elegant Code

Know someone whose code is a bit sloppy or too vibey? This video might be just what they need.

ChatGPT's Atlas Web Browser

Yesterday, ChatGPT released a standalone Web browser, called Atlas, with ChatGPT integrated into it. You can download it here for macOS only. Other operating systems are coming soon. 

As a web browser, it seems OK. 
But it really shines when you're on a web page, like a specific restaurant on Yelp, and ask it how long it'll take you to drive there.
"What time should I leave to arrive here by 6:15 today, accounting for live traffic updates?"

Also, since you're logged into your own ChatGPT account, your Atlas website queries are saved in ChatGPT. 

Perhaps, one day, Apple will integrate AI into their operating systems so it can answer questions like this based on what you're looking at on your screen.