Guten Tag! 👋 

Many greetings from Munich, Germany.

Last week we extracted data from one PDF with {ellmer}. But often we don’t want to get data out of one single PDF. Instead, we want to use multiple PDFs. Today, let’s combine {ellmer}, {fs}, {purrr} and {pdftools} to do that in one smooth pipeline. 🥳 

Getting the PDF paths

As you know, the {fs} package allows you get paths to many files in a directory. For example, I have a couple of PDFs inside a pdfs/ directory:

As you can probably guess from these paths, these are the PDF penguin reports that we generated a while back. Back then, I used Quarto & Typst. But nowadays I just use Typst. In any case, that’s how one of the PDFs looks:

Get the text content from all PDFs

Next, we can get the text content from all PDFs. The magical {purrr} package makes it really easy to apply the pdftools::pdf_text() function iteratively. That’s perfect to iterate over the paths_pdf vector we’ve just created.

This looks pretty ugly. So let’s remove all extra white spaces with stringr::str_squish(). Once again, we can use {purrr} to iterate over all those texts.

Ahh that looks much better.

Do many LLM calls

Now, you might think that we can also use {purrr} to iterate over all these pages. And you’re right. We could. But that’s not the efficient way to do it.

You see, {ellmer} allows us to run data extractions in parallel. We just have to pass a list of prompts to parallel_chat_structured(). Here’s how that looks:

Sweet! With that we have looped over our entire content and extracted information from it. Even better, this was much faster than running 3 LLM calls one after the other. With parallel_chat_structured(), you will asynchronously sent a bunch of llm calls all at once.

Even better: This happens all in the same R process. So you didn’t even have to think about async stuff like the things that the {mirai} package allows you to do. And {ellmer} will even take care of sending LLM calls in bits and pieces. Let me demonstrate that with the progress bar that you’ve seen above. I’ll simply repeat a bunch of my data and send more requests to the LLM:

Have a look how the progress bar evolves. Row-by-row {ellmer} will tell you that:

• 15 LLM calls are queued, 0 have to be retried, 9 calls are waiting on a reply and 6 already finished

• 0 queued, 0 retries, 10 still waiting, 20 finished

• 0 queued, 0 retries, 1 still waiting, 29 finished

• All done

So as you can see we can throw a whole bunch of data at {ellmer} and it will process that all in parallel for us. Pretty sweet, isn’t it?

Alrighty, that was everything I wanted to cover this week. Hope you enjoyed it. Feel free to hit the reply button to tell me about how you liked this newsletter episode.

And don’t forget about the “Report Automation With Typst” course. Offer is still available until Sunday.

See you next week,
Albert

Whenever you’re ready, there are three I can help you:

1. Automate Your Data Reports: This course helps data analysts eliminate manual copy-paste reporting by automating PDF reports end-to-end, saving hours every cycle and preventing costly mistakes. (Using the lovely Typst language 😍)

2. Generate Insights in Minutes, not Hours: This comprehensive course teaches you to handle data faster, smarter, and more efficiently.

3. Bespoke Data Science Solutions: I’ve helped clients build their own data science solutions. Whether building custom web apps, PDF reports, AI automations or teaching workshops, I’ve got you covered. You can reach out to me via this form (or simply hit reply to this email)

Guten Tag! 👋 

Many greetings from Munich, Germany.

Last week I gave a talk in front of like 50 actuaries who mostly deal with Excel to wrangle data. My task was to show them the tools of the trade that data scientists like to use to wrangle data. So naturally this meant talking about programming to an audience that isn’t too keen about programming (as they mostly use Excel.)

Luckily for me I had a special treat with me. You see, I also had slides about Typst. Now, you may wonder why would a non-programmer audience suddenly be excited about Typst!?

But here’s the thing: I didn’t just talk about Typst. What I actually talked about was automating reports using Typst combined with R or Python.

As you’d expect, everyone in the room was familiar with the process of

• crunching numbers,

• copying that over to Word,

• and then printing that as a PDF.

And of course everyone knows how annoying and error prone this process is.

So that’s why I could see so many people become excited about the promise to automatically generate reports from data. Especially after I showed them that this PDF is also generated automatically:

That’s the power of Typst. Not only can it generate PDFs, it can also baffle a non-programmer audience with its skills for working with data and automating tedious processes. How cool is that?

So that’s why I’m super excited to share that I have finally recorded the video that walks you through how to generate the above PDF. The video will be edited later this week.

That’s why this announcement is a heads-up: I am about to take my “Report Automation With Typst” course from pre-sale mode into its final form.

This will also remove the pre-sale discount that is currently available. So you can still get the full course at pre-sale price before I increase the price on Sunday, July 12.

And even better: I’m adding a “it’s finally done” discount to the current pre-sale price. Use the code “ITSDONE10” at checkout to get 10% off the pre-sale price (meaning the discount will also be gone on Sunday.)

If you’ve been on the fence about the course, this is the moment to jump in. Once the video is live and the price goes up, this deal is gone for good. So if you’re ready to automatically generate PDFs from data, you can learn how to do that here:

Best,
Albert

Whenever you’re ready, there are three I can help you:

1. Automate Your Data Reports: This course helps data analysts eliminate manual copy-paste reporting by automating PDF reports end-to-end, saving hours every cycle and preventing costly mistakes. (Using the lovely Typst language 😍)

2. Generate Insights in Minutes, not Hours: This comprehensive course teaches you to handle data faster, smarter, and more efficiently.

3. Bespoke Data Science Solutions: I’ve helped clients build their own data science solutions. Whether building custom web apps, PDF reports, AI automations or teaching workshops, I’ve got you covered. You can reach out to me via this form (or simply hit reply to this email)

Guten Tag! 👋 

Many greetings from Munich, Germany. Last time we talked about {ellmer} and chat objects. Now, let's figure out how to do work with documents. You know, so that we can do AI-assisted document processing.

Unfortunately, data scientists are often great at wrangling tabular data like tibbles or dataframes. But data scientists don't necessarily know how to deal with file formats. That was one reason why my data cleaning masterclass had one part solely focused on how to deal with different file formats. Still, that focused on files containing tabular data.

In our new scenario, things are a bit different. We don't want to get a tabular file format into an LLM. Instead, we want to work with arbitrary text files like PDFs or Word documents or other file formats. And there are two ingredients for this. First, there is the file management part and then there’s the text extraction part. Let's talk about them both today.

File management

There are many functions inside R that allow you to manage files. But I feel like the {fs} package gives you a more consistent interface. Basically, the functions come in three flavors for files, paths and directories. The functions start with file_* , path_* or dir_* respectively:

Examples mostly taken from the {fs} documentation

This allows you to manage files effectively through a consistent, unified interface. And yes, I know how incredibly boring this sounds. But it becomes really convenient when you have to deal with many files. So, if you want to learn more about that, I’ve released a video that goes into more details about {fs}:

Text Extraction

Once you know how to manage files, you can extract texts from them. For most use cases, you’ll need texts from PDF. For that, there is the {pdftools} package. If we had an invoice stored at dat/invoice.pdf we could access the text like so:

For Word documents, you could use the {officer} package. It allows you to first extract the content elements in a structured form:

And if you filter that for paragraphs, then you’d get only the texts:

Of course, once you have the raw text, you can apply one or more of the many text cleaning functions that the {stringr} package provides you. That's particularly useful if you're looking to add or subtract specific information that can be encoded programmatically before passing the content to an LLM. Feel free to check out the {stringr} docs or my Data Cleaning Master Class for more on text cleaning.

That was it for today. For simple text extractions, you've got the tools you need now. If you want to go even further, you could do an OCR using a service like Azure Document Intelligence or AWS Textract. But that’s a story for some other time.

Best,
Albert

Whenever you’re ready, there are three I can help you:

1. Automate Your Data Reports: This course helps data analysts eliminate manual copy-paste reporting by automating PDF reports end-to-end, saving hours every cycle and preventing costly mistakes. (Using the lovely Typst language 😍)

2. Generate Insights in Minutes, not Hours: This comprehensive course teaches you to handle data faster, smarter, and more efficiently.

3. Bespoke Data Science Solutions: I’ve helped clients build their own data science solutions. Whether building custom web apps, PDF reports, AI automations or teaching workshops, I’ve got you covered. You can reach out to me via this form (or simply hit reply to this email)

Guten Tag! 👋 

Many greetings from Munich, Germany. I see more and more data scientist being thrown into loosely data-related activities. The reason for that is (of course) AI. Ugh.

Yes, AI is the gift that keeps on giving. And if you're anything like me, you probably have a love and hate relationship with AI. It is such a super cool tool and I use it every day. But the hype around it is just maddening.

Aaaannnyway, the reason for why I'm writing about AI here is because data people are being thrown into the deep engineering end.

“It's related to data”, they said. “So it must be a good task for data scientists”, they said.

Well, over the last three years I’ve been building AI-powered automations. You know, the (by now) classical automation that

• takes in a document,

• extracts all the data that is desired and then

• passes the structured data along to clients or some other part of the company.

That is a cool thing to work on. I love the automations that come out of this.

But time and time again, I realized that the tasks that come with building such an automation are not really what data scientists learn in their training.

But don’t fret. Let’s just change that.

Over the next couple of weeks I want to give you a high level overview of what it can entail to bring an AI document extraction into production.

Revisiting the AI basics

If you’ve been a long time reader, you know that long ago I published a little Quarto book called “R with AI”. In there, I explained how to use the {ellmer} package to use AI from R. And just like meeting the love of your life, it can all start with a simple chat:

Unlike the love of your life, AI keeps asks you for an API key. So the above code only works if you have an ANTHROPIC_API_KEY environment variable. With this chat, you can call one of its methods to talk to the AI. Like so:

So how about structured data extraction?

Luckily for us, ellmer also makes it very easy to extract data from the text we give it. All we have to do is use the chat_structured() method of the chat. This in combination with

• some text to extract from and

• a specification of what to extract

will get the job done. The latter is done by chaining/nesting many type_*() functions to establish elaborate structures to extract. Here, we’re just going to extract all the numbers we can find by combining type_number() with type_array() 

Note that the description that you see inside type_*() functions are the prompts for the things you want to extract.

Lets do one more

By combining type_array() and type_object() we can let the AI extract data.frames. In this case, it allows us to avoid regular expressions:

Cool. So AI allows us to avoid writing regular expressions. Revolutionary stuff!

But in all serious, this is a pretty powerful mechanism. So it’s no wonder that I got to spent the last couple of years building automations around this stuff. More on that next time.

Best,
Albert

Whenever you’re ready, there are three I can help you:

1. Automate Your Data Reports: This course helps data analysts eliminate manual copy-paste reporting by automating PDF reports end-to-end, saving hours every cycle and preventing costly mistakes. (Using the lovely Typst language 😍)

2. Generate Insights in Minutes, not Hours: This comprehensive course teaches you to handle data faster, smarter, and more efficiently.

3. Bespoke Data Science Solutions: I’ve helped clients build their own data science solutions. Whether building custom web apps, PDF reports, AI automations or teaching workshops, I’ve got you covered. You can reach out to me via this form (or simply hit reply to this email)

Guten Tag! 👋 

Many greetings from Munich, Germany. In my last mail, I announced that I'm working on a workshop to bring more technical content into the R community.

And wow. Many of you had thoughts on that.

Turns out, I hit a nerve when I said that there are too few resources on bringing R into production. But not in a bad way. Your responses were more like “Yes, exactly! This is what I've been looking for.”

So I'm glad that I wasn't imagining things. And I'm also glad that there's an interest in this community to close many of these technical gaps.

Not surprisingly, most responses were about two things:

1. How to work (more efficiently) with databases and cloud storage.

2. How to use functional and functional object oriented programming to write more robust R code.

And this doesn't surprise me because both things are real struggles that you hit immediately when working in a business setting. Also, the fact that there's a high demand for such a fundamental building block only underlines my initial assessment: R users are often left alone to figure out how to operate their favorite language in business settings.

So this only strengthens my resolve to teach an online workshop on some of these more technical topics. Details to follow. In the meantime, you can check out my newest YT video on how to connect your favorite data science language to databases, cloud storage or APIs:

Best,
Albert

Whenever you’re ready, there are three I can help you:

1. Automate Your Data Reports: This course helps data analysts eliminate manual copy-paste reporting by automating PDF reports end-to-end, saving hours every cycle and preventing costly mistakes.

2. Generate Insights in Minutes, not Hours: This comprehensive course teaches you to handle data faster, smarter, and more efficiently.

3. Bespoke Data Science Solutions: I’ve helped clients build their own data science solutions. Whether that’s building custom web apps, PDF reports, AI automations or teaching workshops, I’ve got you covered. You can reach out to me via this form (or simply hit reply to this email)

Guten Tag!

Many greetings from Munich, Germany. In today’s newsletter, I want to teach you about database transactions. Sounds simple, but it’s actually a very crucial concept that we haven’t covered before.

You see, in many cases you’ll want to write to your database using transactions. Technically, you could just write to your database as-is. But this leaves you without any safety precautions in case something goes wrong.

For example, you could lose your internet connection in the middle of uploading a large dataset. If that happens your database ends up with only part of the data, and then you’re stuck trying to figure out:

• What was successfully transferred?

• What’s still only on your local machine?

These kinds of situations can easily leave your database in a broken or inconsistent state. And that’s really really frustrating to deal with.

So that’s where transactions come in. Let’s dive in.

What Are Transactions?

Transactions allow you to tell your database ahead of time

“Hey, I’m going to write something. I’ll let you know when I’m done. Don’t finalize anything until I say so.”

If everything works fine, you “commit” the transaction, and the data gets saved. If not, you “roll it back,” and it’s like nothing ever happened.

Sounds simple, right? Well, it is. You just have to use this nifty little safety feature. Your database will thank you.

How to Use Transactions in R

It’s surprisingly easy to use transactions in R when working with databases. You start a transaction with:

Then, you perform your usual data writing operations. Let’s assume we’re just going to add a few rows:

If everything goes well, you finalize the changes with:

Or if anything goes wrong, you can cancel the whole transaction, i.e. everything you did since you called dbBegin():

Why Is This Helpful?

This is especially useful when you’re working with multiple tables.

Let’s say you need to insert data into one table and then into another table that references the first table. Let’s try this:

Notice how the second call failed? That happened because I put restrictions on the second table whereas the first one doesn’t have them.

Either way, that’s when we call dbRollback(). And in order for all of this to happen automatically, we can wrap the whole procedure into a tryCatch() statement.

And with that, you’ve got a new set of tools to interact with databases safely and reliably. Use transactions whenever you write to a database, especially when multiple related operations are involved. Your future self (and your data) will thank you!

And as always, if you have any questions, or just want to reach out, feel free to contact me by replying to this mail or finding me on LinkedIn or on Bluesky.

See you next week,
Albert 👋

Enjoyed this newsletter? Here are other ways I can help you:

Data Cleaning With R Master Class

A data scientist’s guide to avoid wasting time with data cleaning. Learn 5 fundamental aspects of data cleaning, get to your insights much quicker.

data-cleaning.albert-rapp.de

Insightful Data Visualizations for "Uncreative" R Users

In this course, I teach you how to create insightful data visualizations without being a design expert.

arapp.thinkific.com/courses/insightful-data-visualizations-for-uncreative-r-users

Making Beautiful Tables with R

Tables are often boring, hard to read, and, most importantly, they don't communicate effectively. In this course, I will show you that creating beautiful tables is just a matter of learning a few important principles.

rfortherestofus.com/courses/tables

• My playlist on YouTube

• Free book: Yet Again - R + Data Science

• Free book: Beautiful tables with {gt}

• Blog posts

Guten Tag!

Many greetings from Munich, Germany. The glue_sql() function helped us last week to make our database calls safer and avoid SQL injection.

But there’s also a lot of convenience that glue_sql() offers for everyday development. So let’s check that out.

Use Curly Braces for R Variables

The first thing that’s really convenient and that you already know from last week is this: You can use curly braces {} inside your SQL string to insert R variables. This works great when the variables are just simple strings or scalar values.

And as you can see, this gives you the results you probably want:

Use Backticks for IDs

Next, let’s try to not grab all columns but only a selected one:

Well, that didn’t crash but it didn’t go as expected either. The reason for that is that you will have to tell glue_sql() that col_name (and technically also tbl_name) is an ID. You can do that with backticks inside the curly brackets.

Ahh this looks much better!

Collapse Vectors with the * Operator

If you pass a vector into a placeholder, glue_sql() by default tries to insert the whole vector resulting in multiple SQL fragments:

This is usually not what you want. Instead, you want to collapse a vector with *:

Sweet! With that you can easily get data from multiple columns:

Use IDs from different tables

Let’s imagine that our database has another table:

Then, we can make sure to reference to columns from different tables by including the table names. The Id() function helps with that:

And because we need to identify the corrsponding species columns somehow, we have to reference to the corresponding table names as well.

Now, we could easily wrap all of these IDs into backticks and everything works out nicely in assembled SQL code. Here’s an example from the glue_sql() docs for that:

And there you have it: glue_sql() will make your life as an R developer much easier. Beyond just protecting you from SQL injection, it helps you write cleaner, more flexible, and maintainable database queries from within R.

And as always, if you have any questions, or just want to reach out, feel free to contact me by replying to this mail or finding me on LinkedIn or on Bluesky.

See you next week,
Albert 👋

Enjoyed this newsletter? Here are other ways I can help you:

Data Cleaning With R Master Class

A data scientist’s guide to avoid wasting time with data cleaning. Learn 5 fundamental aspects of data cleaning, get to your insights much quicker.

data-cleaning.albert-rapp.de

Insightful Data Visualizations for "Uncreative" R Users

In this course, I teach you how to create insightful data visualizations without being a design expert.

arapp.thinkific.com/courses/insightful-data-visualizations-for-uncreative-r-users

Making Beautiful Tables with R

Tables are often boring, hard to read, and, most importantly, they don't communicate effectively. In this course, I will show you that creating beautiful tables is just a matter of learning a few important principles.

rfortherestofus.com/courses/tables

• My playlist on YouTube

• Free book: Yet Again - R + Data Science

• Free book: Beautiful tables with {gt}

• Blog posts

Guten Tag!

Many greetings from Munich, Germany. In the last couple of newsletters, we talked about SQL databases. And there’s no way we can talk about accessing SQL databases via R code without mentioning SQL injection.

This is a nasty thing that can leak data or even destroy your database if you allow people to interact with it unsafely. And often, that’s exactly what you want (the interaction, not the unsafe part 😆). Here’s a common scenario for that:

• Users interact with a Shiny dashboard to retrieve specific data from a database.

• Based on the inputs, you execute some SQL.

• Your dashboard displays the new data.

So letting users interact with a database is not bad in itself. It’s only bad when you do it uncontrolled. That’s when this bad SQL injection can occur.

But here’s the good news: There’s an easy way to avoid damage. And even better: This method also makes it much easier to assemble SQL code in R.

Let me show you how that works.

SQL Injection via R Variables

Let’s talk about how you might encounter SQL injection in the first place.

Let’s imagine that you want to retrieve data about a specific penguin species from a table in your database. One common (but unsafe) way to do this is to paste together SQL code manually with R variables:

While this works, it’s insecure. If a user-supplied value like species contains malicious SQL code, it could complete your SQL snippet and do something dangerous (e.g. extracting all of your confidential data). This is SQL injection.

And in this specific scenario, here’s what that would look like. Let’s first set up our data base from last time:

And then we can stick in some malicious code into the species variable.

Here, a little trick was used to close the “WHERE” statement with a ' followed by an or-statement that is always true. That way, we will never actually filter the data.

Why Would That Happen?

Now, you might wonder: Why would my variable ever be filled with something malicious?

Well, consider a Shiny app that allows users to filter database results. If you’re not careful, users can input harmful values into the fields that populate those SQL variables. If someone with bad intentions figures out you’re pasting user input into SQL strings, they could inject destructive SQL.

And they don’t even have to do that manually. There are automated tools that try out loads of combinations in the hopes that at some point some SQL injection works.

The Solution: glue_sql() from {glue}

Here’s where {glue} and its glue_sql() function come in. Using glue_sql() makes assembling SQL both easier and safer. You still write your SQL code in text form, but you use curly braces ({}) to insert variables.

Most importantly, glue_sql() handles SQL escaping for you to prevent SQL injection. Here’s an example:

So in the standard case everything works like you’d expect. And in case there’s the same malicious code from before inside the species variable?

Here, glue_sql() will put everything into quotes. That way, a user cannot stick in something to close the initial quotation mark. And the resuls would be empty:

Thus, this is easy to write and secure.

Fantastic! We’ve just seen how glue_sql() helps you write safer SQL code. But there’s more! In the next newsletter, we’ll explore additional features of glue_sql() that make it even more powerful.

And as always, if you have any questions, or just want to reach out, feel free to contact me by replying to this mail or finding me on LinkedIn or on Bluesky.

See you next week,
Albert 👋

Enjoyed this newsletter? Here are other ways I can help you:

Data Cleaning With R Master Class

A data scientist’s guide to avoid wasting time with data cleaning. Learn 5 fundamental aspects of data cleaning, get to your insights much quicker.

data-cleaning.albert-rapp.de

Insightful Data Visualizations for "Uncreative" R Users

In this course, I teach you how to create insightful data visualizations without being a design expert.

arapp.thinkific.com/courses/insightful-data-visualizations-for-uncreative-r-users

Making Beautiful Tables with R

Tables are often boring, hard to read, and, most importantly, they don't communicate effectively. In this course, I will show you that creating beautiful tables is just a matter of learning a few important principles.

rfortherestofus.com/courses/tables

• My playlist on YouTube

• Free book: Yet Again - R + Data Science

• Free book: Beautiful tables with {gt}

• Blog posts

Guten Tag!

Many greetings from Munich, Germany. Last week, I showed you how to connect to databases with R. And I also mentioned that it’s helpful to write a bit more SQL code than just grabbing all the data and processing it locally. This might sound like a lot of effort, though. So here’s the good news:

If you’re already using dplyr for data wrangling, you can get the best of both worlds. You can write dplyr code and let R translate that into SQL so that time-saving and resource-efficient calculations happen in the database. Then, you only retrieve the results.

So in today’s newsletter I’ll show you how that works. Let’s go!

Getting Started with {dbplyr}

First things first: you’ll need to install the {dbplyr} package. This package extends dplyr so it can translate R code into SQL syntax for supported databases.

To get started, establish a connection to your database using DBI::dbConnect(), just like before. This part hasn’t changed.

Working with Tables Using {dbplyr}

Now you can use {dbplyr} functions to interact with the database using familiar dplyr verbs. For example, in our example database, you can copy an R dataset like palmerpenguins::penguins into a database table using copy_to():

Fun fact: Notice that I never loaded {dbplyr} 😮 {dplyr} will detect when you want to use a {dbplyr} function.

Once the data is in the database, you can reference it with tbl(). This will connect to the table in the database (which is why the con variable is needed.)

Notice how the output almost looks like a regular tibble. The only difference is that it shows only a preview and not the full dataset. And in case you’re wondering: That’s a good thing. You never know how large the database is and often a preview will suffice.

Click to Support 👇️ 

Hey friends 👋 I’m experimenting with ads in my newsletter to finance all the free stuff I give away. If you enjoy this newsletter and want to support it, a click on the the ad below would be a super simple but effective way to do so. 🤗 

No need to sign up. A click is enough support 🥳 

---

You’ve never experienced business news like this.

Morning Brew delivers business news the way busy professionals want it — quick, clear, and written like a human.

No jargon. No endless paragraphs. Just the day’s most important stories, with a dash of personality that makes them surprisingly fun to read.

No matter your industry, Morning Brew’s daily email keeps you up to speed on the news shaping your career and life—in a way you’ll actually enjoy.

Best part? It’s 100% free. Sign up in 15 seconds, and if you end up missing the long, drawn-out articles of traditional business media, you can always go back.

Check it out

---

Alright, now back to the good stuff.

Writing SQL Through dplyr Verbs

Now that you have a table-like object, you can use familiar dplyr verbs to generate SQL queries. For example:

Again, this only shows a preview. To pull the full result set from the database, use collect().

Seeing the Generated SQL

And if you’re ever wondering how {dbplyr} translated your dplyr code into SQL, you can inspect it using show_query().

Also, this is a great way to learn more about SQL by seeing how familiar R code maps to SQL commands.

Nice. We’ve learned a simple way to interact with databases using familiar syntax. However, using {dbplyr} doesn’t mean you’ll never need to write raw SQL again.

It helps you get started quickly and efficiently, though. Next week, I’ll show you how to combine {dbplyr} with raw SQL for more advanced database workflows.

And as always, if you have any questions, or just want to reach out, feel free to contact me by replying to this mail or finding me on LinkedIn or on Bluesky.

See you next week,
Albert 👋

Enjoyed this newsletter? Here are other ways I can help you:

Data Cleaning With R Master Class

A data scientist’s guide to avoid wasting time with data cleaning. Learn 5 fundamental aspects of data cleaning, get to your insights much quicker.

data-cleaning.albert-rapp.de

Insightful Data Visualizations for "Uncreative" R Users

In this course, I teach you how to create insightful data visualizations without being a design expert.

arapp.thinkific.com/courses/insightful-data-visualizations-for-uncreative-r-users

Making Beautiful Tables with R

Tables are often boring, hard to read, and, most importantly, they don't communicate effectively. In this course, I will show you that creating beautiful tables is just a matter of learning a few important principles.

rfortherestofus.com/courses/tables

• My playlist on YouTube

• Free book: Yet Again - R + Data Science

• Free book: Beautiful tables with {gt}

• Blog posts

Guten Tag!

Many greetings from Munich, Germany. When you’re working in a company, chances are you won’t be handed a simple CSV file to work with. Instead, you’ll likely need to grab your data from a company-wide SQL database hosted in the cloud or on internal servers.

So in today’s newsletter, I’ll show you how to connect to these databases using R.

What The Heck is a Driver?

First, we need a so-called driver. In simple terms, this is the tool that handles the connection between your computer and the database. Think of it as a translator that often comes in one of two flavors:

• ODBC drivers: These drivers work with the open database connection format

• JDBC drivers: These drivers use Java (that’s what the “J” stands for in JDBC)

Here, we’ll use the {odbc} package to go down the ODBC route.

Which Driver Do You Need?

So now that we know that we want ODBC drivers, we also need to decide which specific driver we need. You see, this kind of thing depends on the database your or your company uses. For example it could be an Oracle or PostgreSQL database that will need an Oracle or PostgreSQL driver

Typically, your tech department will tell you what you need. Your job is only to understand the big picture that I’m trying to explain here.

Example: Setting Up an SQLite Database

For today’s example, we’ll simulate a database using SQLite, which supports in-memory databases. This means that we can simply spin up a temporary database locally on our computer.

After having installed a SQLite driver for your computer, you can set up a connection from R to a temporary database using {DBI} and {odbc}:

Whooah. That’s a lot of function arguments. Once again, the connection details (like database name, user, password and port) is something that your tech department should have ready for you.

Creating and Populating a Table

So now that we have a database connection stored in a variable called con, we can create a table in our SQLite database. Here, we’ll use the famous Palmer Penguins dataset:

Querying Data from the Database

To fetch data, you’ll use dbGetQuery() from the {DBI} package. Here’s how to select all rows using SQL code:

Alternatively, you can (and probably should) use a bit more SQL to filter down the data that you actually want.

Why Use SQL for Filtering?

And you may wonder: “Why should I invest time into learning SQL when I can just grab the full data using the snippet you’ve just shown?“ And indeed, you could download the entire table and filter it inside of R. But filtering directly in the database is often much faster because:

• SQL databases are optimized for these operations

• You avoid transferring large datasets over the network

Learning a few basic SQL keywords can go a long way in improving your data workflow.

Nice! With that we covered:

• The basics of database connections in R

• The difference between ODBC and JDBC

• How to set up and query a database using {DBI} and {odbc}

Armed with these skills, you’ll hopefully be ready to connect to real company databases and work with large-scale data. As always, if you have any questions, or just want to reach out, feel free to contact me by replying to this mail or finding me on LinkedIn or on Bluesky.

See you next week,
Albert 👋

Enjoyed this newsletter? Here are other ways I can help you:

Data Cleaning With R Master Class

A data scientist’s guide to avoid wasting time with data cleaning. Learn 5 fundamental aspects of data cleaning, get to your insights much quicker.

data-cleaning.albert-rapp.de

Insightful Data Visualizations for "Uncreative" R Users

In this course, I teach you how to create insightful data visualizations without being a design expert.

arapp.thinkific.com/courses/insightful-data-visualizations-for-uncreative-r-users

Making Beautiful Tables with R

Tables are often boring, hard to read, and, most importantly, they don't communicate effectively. In this course, I will show you that creating beautiful tables is just a matter of learning a few important principles.

rfortherestofus.com/courses/tables

• My playlist on YouTube

• Free book: Yet Again - R + Data Science

• Free book: Beautiful tables with {gt}

• Blog posts

Guten Tag!

Many greetings from Munich, Germany. Environment variables control system-wide or session-wide settings inside your R session. This sounds incredibly boring. And, honestly, it kind of is.

But it can feel complicated to R users who haven’t encountered them before. When you do bump into environment variables “in the wild,” they can seem unnecessarily tricky.

That’s why, in today’s newsletter, I’ll cover a bit of the mundane stuff. That is, I’ll show you how to set and get environment variables in R. I know: Exciting! So let’s dive in.

When Do Environment Variables Matter?

Let’s quickly look at a couple of real-world examples where environment variables show up in R workflows:

Using APIs (like OpenAI)

If you’re working with APIs, such as using the {ellmer} package to connect to ChatGPT or other AI services, you’ll need to store your API keys as environment variables.

For example, to use OpenAI’s API, you need to specify the environment variable OPENAI_API_KEY. This keeps your credentials secure and avoids hard-coding sensitive infos into your scripts.

Deploying Shiny Apps

If you’re deploying a Shiny app, you might want to keep different settings for:

• Production (the live version of your app that your users see)

• Development (the version of the app that you as a developer see and use to test new features)

Environment variables let you switch configurations easily depending on your deployment stage (without editing the core code.)

So with the examples out of the way, let’s dive into three popular methods for setting environment variables.

Method 1: .Rprofile (My Favorite)

The .Rprofile file allows you to run R code automatically at the start of each R session. If your project doesn’t have such a file, just create a text file and name it .Rprofile. And to set an environment variable, you can use Sys.setenv() inside the .Rprofile file:

🚨 NOTE: Only use Sys.setenv() inside your .Rprofile. If you put it in a regular script, it might get saved to your R history file. And if that ever gets pushed to a public repo, your API key could be compromised.

Method 2: .Renviron

The .Renviron file is simpler. Unlike .Rprofile, it cannot execute R code as it only holds key-value pairs. But just like .RProfile, these environment variables are always loaded at the start of an R session.

The file itself always follows the syntax key=value. Here’s an example:

Method 3: .env (via {dotenv})

The .env file is popular in many programming languages (like Python). A .env file looks just like .Renviron. The difference is only that you load the .env file manually. In R, you can use the {dotenv} package to do that.

This method is especially handy if you want to share environment variables across R and Python.

There you have it. Three ways to work with environment variables in R:

1️⃣ .Rprofile (using Sys.setenv())
2️⃣ .Renviron (key-value only)
3️⃣ .env (loaded via {dotenv})

Sure, it’s not the flashiest topic but it’s super important once you move into production-like environments or need to store secrets. As always, if you have any questions, or just want to reach out, feel free to contact me by replying to this mail or finding me on LinkedIn or on Bluesky.

See you next week,
Albert 👋

Enjoyed this newsletter? Here are other ways I can help you:

Data Cleaning With R Master Class

A data scientist’s guide to avoid wasting time with data cleaning. Learn 5 fundamental aspects of data cleaning, get to your insights much quicker.

data-cleaning.albert-rapp.de

Insightful Data Visualizations for "Uncreative" R Users

In this course, I teach you how to create insightful data visualizations without being a design expert.

arapp.thinkific.com/courses/insightful-data-visualizations-for-uncreative-r-users

Making Beautiful Tables with R

Tables are often boring, hard to read, and, most importantly, they don't communicate effectively. In this course, I will show you that creating beautiful tables is just a matter of learning a few important principles.

rfortherestofus.com/courses/tables

• My playlist on YouTube

• Free book: Yet Again - R + Data Science

• Free book: Beautiful tables with {gt}

• Blog posts

Guten Tag!

Many greetings from Munich, Germany. In today’s newsletter, I’ll show you how to integrate multiple parametric report sections into a single report. You see, instead of generating separate reports for each dataset, you can include multiple sections where each sections uses different data.

This could look something like this:

• 📄 General intro text (applies to all datasets)

• 🐟 Section A (Dataset A)

• 🐦 Section B (Dataset B)

• 🦋 Section C (Dataset C)

• ✅ General conclusion

So let’s see how to build this.

Step 1: Start with a Parametric Report

Begin with a standard parametric report, just like you normally would.

For example, let’s say you have the Quarto file from last time that generates a report for a single species. This was driven by a species and a color parameter in the YAML header:

This report works fine on its own, but we want to reuse the code and text within the report multiple times in a single document.

Step 2: Create a Main Quarto File

Next, create a new Quarto file that will act as the main document. This file will include first some general intro text, followed by multiple “includes” of your parametric report. For example, the document could start like this:

Step 3: Use include Snippets

Quarto has a special feature that lets you include other Quarto files using the following syntax:

Here, this will include text and code chunks from the Quarto file species_report.qmd. That way, you could modify your main Quarto file like this:

But if you render this now, you’ll hit an error that will say that params is not defined. If you recall, params is the object that we use inside the species Quarto file.

Normally, this object is created from the parameters that are specifies inside the YAML header. But here this doesn’t work because includes don’t copy the YAML header from the included file.

Step 4: Define params in the Master File

To solve this, define the params object manually in a code chunk before each include. For example, your report could look like this:

That way, the copied-and-pasted code from species_report.qmd will work because it is preceded by a code chunk that defines params.

Step 5: Repeat for Multiple Datasets

Now, you can repeat the same process to include multiple sections, each with a different dataset:

This reuses the same species report over and over, but with different params each time. Super efficient, isn’t it?

This approach allows you to

• reuse the same report structure,

• render multiple sections with different data, and

• produce a single, polished document

It’s a powerful pattern when working with multiple datasets that need similar analysis! Pretty neat, right? As always, if you have any questions, or just want to reach out, feel free to contact me by replying to this mail or finding me on LinkedIn or on Bluesky.

See you next week,
Albert 👋

Enjoyed this newsletter? Here are other ways I can help you:

Data Cleaning With R Master Class

A data scientist’s guide to avoid wasting time with data cleaning. Learn 5 fundamental aspects of data cleaning, get to your insights much quicker.

data-cleaning.albert-rapp.de

Insightful Data Visualizations for "Uncreative" R Users

In this course, I teach you how to create insightful data visualizations without being a design expert.

arapp.thinkific.com/courses/insightful-data-visualizations-for-uncreative-r-users

Making Beautiful Tables with R

Tables are often boring, hard to read, and, most importantly, they don't communicate effectively. In this course, I will show you that creating beautiful tables is just a matter of learning a few important principles.

rfortherestofus.com/courses/tables

• My playlist on YouTube

• Free book: Yet Again - R + Data Science

• Free book: Beautiful tables with {gt}

• Blog posts

Guten Tag!

Many greetings from Munich, Germany. In this week’s newsletter, we’re continuing our journey with Typst + Quarto. As promised, we’re taking our PDF reports to the next level with some visual polish.

Today, I want to show you how to:

1. Add a vertical color bar along the left margin of every page

2. Wrap your main title in a colored box and shift it slightly to the left for a bold visual effect

These small touches can improve the look and feel of your report. And once you know how these things work, you can work out how to add more visual elements with Typst.

Add a Vertical Color Bar to the Left Margin

Let’s start by giving all pages a distinctive look using a colored stripe along the left-hand side. To do this, we modify the set page() section of our Typst template. You know, this part here:

Here, we can use the background property with the place() function to overlay a rectangle:

And with this simple change, we’ve added a vertical splash of color to every page.

Style the Main Title with a Colored Box

Next, let us give our report title more visual weight. Instead of plain text, we’ll wrap it in a colored box. We do so by wrapping our text() function into a box() function. That way, our previous pages specification

becomes

Now, our document looks like this:

Shift the title box

Finally, we can shift the title box slightly to the left by adding a bit of negative horizontal space.

With just a few small tweaks, we now have a PDF report that is not only parameterized with different inputs but it also looks polished and professional.

And as always, if you have any questions, or just want to reach out, feel free to contact me by replying to this mail or finding me on LinkedIn or on Bluesky.

See you next week,
Albert 👋

Enjoyed this newsletter? Here are other ways I can help you:

Data Cleaning With R Master Class

A data scientist’s guide to avoid wasting time with data cleaning. Learn 5 fundamental aspects of data cleaning, get to your insights much quicker.

data-cleaning.albert-rapp.de

Insightful Data Visualizations for "Uncreative" R Users

In this course, I teach you how to create insightful data visualizations without being a design expert.

arapp.thinkific.com/courses/insightful-data-visualizations-for-uncreative-r-users

Making Beautiful Tables with R

Tables are often boring, hard to read, and, most importantly, they don't communicate effectively. In this course, I will show you that creating beautiful tables is just a matter of learning a few important principles.

rfortherestofus.com/courses/tables

• My playlist on YouTube

• Free book: Yet Again - R + Data Science

• Free book: Beautiful tables with {gt}

• Blog posts

Guten Tag!

Many greetings from Munich, Germany. Have you ever heard of the Typst project? It is a lightweight, Markdown-inspired language designed for creating beautiful, customized PDF documents with minimal effort.

Basically, it is supposed to be like LaTeX but muuuch easier. And the coolest part? You can use Typst from within Quarto to create stunning PDF reports.

So in today’s newsletter, I’ll walk you through how to style our PDF document from last week. Let’s dive in

Set Typst as Your PDF Format

To start using Typst from Quarto, go into your .qmd file and specify typst as the output format in the YAML header:

If you render this now, you’ll get a similar PDF output as before.

Add Custom Styling with Typst Templates

To make our PDF look good, we’ll use template partials provided by Typst. There are two key files that we need to mention in the YAML header for that:

Both are plain text files that you can create by just taking a .txt file and changing the extension to .typ. So the more important question is: “How do we fill these files?” Let’s go through both files.

Filling typst-show.typ

This file defines the variables that are passed into the layout, such as the title, author, and parameters from your Quarto YAML. You see, what you you define in the Quarto YAML header isn’t automatically known to Typst. That’s why you need to transfer it. For that you use the #show function from Typst.

There, you specify that your “article” layout (that’s the default name of the used layout) is equipped with a set of variables.

The syntax takes some getting used to. Basically, you always write

Yeah, I know. As I said, it takes some getting used to. And for drilling down nested YAML structures, you can just use a dot symbol (.). With that we know enough Typst code to transfer title, author and params.species from our Quarto YAML header to Typst.

Filling typst-template.typ

Nice. One file down, one more to go.

In the typst-template.typ file, we define the actual article layout of our PDF report. For that, we use the #let function.

Inside the function body (indicated by {}), we can now set attributes of

• document (contains metadata),

• text (defines text properties of the document), and

• page (defines the properties of every page in the document).

Have a look at the code and see if you can make out what it does. I like to think that it’s somewhat manageable to piece together what it does. But don’t worry. I’ll explain afterwards anyway.

So here, we have used the set statement to specify properties of the document, text and page objects. There, we’ve set some of the objects attributes (like title, font or paper) to values that were

• either hard-coded (like "Source Sans Pro" or the margins) or

• filled using arguments of the #let() function (like author or paper).

Add a page

So far so good. We’ve set a couple of things. But we haven’t added any content to the layout yet. Let’s change that.

Let’s insert a page using the page() function. In there, we can make sure that everything is left-aligned and starts at the top of the page using align(). The syntax for that is

Add content

See that line that says “page content goes” here? That’s where we can place the content of the page. Here, let’s just try to use the body variable from the article() function to insert everything that is inside our Quarto document.

That should be a valid layout. Let’s try to render this. Here’s what we’ll get:

Oh no. That’s a bit dumb. Instead of understanding body as variable, Typst things that we actually want to use a literal text. But we have used other variables just like that before, haven’t we? What’s going on?

Markdown vs evaluation context

You see, we are currently filling the align() function inside the page() function with content. This content is inside brackets [] where it is evaluated as Markdown. Not in any functional kind of way. And the magical signal for Typst to switch to an evaluated context is #.

That’s also why the let() function to define the article layout started with #. Otherwise, Typst would think that our file is just filled with plain text and won’t evaluate anything. And the same has do be done now that we are inside a Markdown content block inside the brackets of align(). So, let’s try this:

Ahh much better. There’s all of our content. And notice how everything fits on one page now.

Add our parameters

Similarly, we can add our parameters using the text() function. The syntax is the same as with align():

• First, we fill text() with arguments like text(size:1.6em).

• Then, we create actual content inside the brackets like `text(size: 1.6em)[content].

• And finally we have to use # inside the Markdown content blocks to use functions and variables.

This will give us this terrible look.

Let’s add some vertical space with the v() function.

And then we have way more space:

Ahh much better. Let that sink in for now. Next week, I’m showing you how to add colors to our designs.

And as always, if you have any questions, or just want to reach out, feel free to contact me by replying to this mail or finding me on LinkedIn or on Bluesky.

See you next week,
Albert 👋

Enjoyed this newsletter? Here are other ways I can help you:

Data Cleaning With R Master Class

A data scientist’s guide to avoid wasting time with data cleaning. Learn 5 fundamental aspects of data cleaning, get to your insights much quicker.

data-cleaning.albert-rapp.de

Insightful Data Visualizations for "Uncreative" R Users

In this course, I teach you how to create insightful data visualizations without being a design expert.

arapp.thinkific.com/courses/insightful-data-visualizations-for-uncreative-r-users

Making Beautiful Tables with R

Tables are often boring, hard to read, and, most importantly, they don't communicate effectively. In this course, I will show you that creating beautiful tables is just a matter of learning a few important principles.

rfortherestofus.com/courses/tables

• My playlist on YouTube

• Free book: Yet Again - R + Data Science

• Free book: Beautiful tables with {gt}

• Blog posts

Guten Tag!

Many greetings from Munich, Germany. Parameterized reports are a powerful but often underutilized strategy for communicating data to multiple stakeholders.

Let’s say you have several departments in your organization. A common instinct is to build a dashboard where users can click around to see the data for their department.

But what if, instead, you could automatically generate one customized report per department and send those out directly? In today’s newsletter, I’ll show you how to do just that using Quarto.

Create a Basic Report Template

First, we need a sample report. We’ll use the palmerpenguins dataset to simulate a situation where each penguin species represents a department.

This will give us a PDF output like this:

Add Parameters to the YAML Header

Here, we’ve used the "Adelie" species to act a placeholder. Next, we have to make sure that we can easily replace the species in all instances of the text and code. To do that, we define a params section in the YAML header:

Now, wherever we want to reference species in the document, we use params$species. This allows the content of the report to adapt to whatever value we plug into the parameter. It works in code chunks:

And it works in inline text as well:

Now we can try to change "Adelie" to "Gentoo" and re-render the report. You’ll see the whole report adjusts accordingly (most notably seen by the plot change.)

Automate Rendering for Multiple Parameters

Now the real power kicks in. We want to render one report per species, or in your real-world scenario, one report per department. To do that, we have to create a separate R script that uses the quarto_render() function from the {quarto} package:

This script loops over all species and renders a separate PDF report for each one.

With just a few lines of configuration and code, you now have a scalable way to generate customized PDF reports for any number of stakeholders or categories. Whether you’re reporting by department, product line, region, or team, parameterized Quarto documents make it simple and powerful.

Next week, we’re going to make that report prettier using the Typst-PDF format. And as always, if you have any questions, or just want to reach out, feel free to contact me by replying to this mail or finding me on LinkedIn or on Bluesky.

See you next week,
Albert 👋

Enjoyed this newsletter? Here are other ways I can help you:

Data Cleaning With R Master Class

A data scientist’s guide to avoid wasting time with data cleaning. Learn 5 fundamental aspects of data cleaning, get to your insights much quicker.

data-cleaning.albert-rapp.de

Insightful Data Visualizations for "Uncreative" R Users

In this course, I teach you how to create insightful data visualizations without being a design expert.

arapp.thinkific.com/courses/insightful-data-visualizations-for-uncreative-r-users

Making Beautiful Tables with R

Tables are often boring, hard to read, and, most importantly, they don't communicate effectively. In this course, I will show you that creating beautiful tables is just a matter of learning a few important principles.

rfortherestofus.com/courses/tables

• My playlist on YouTube

• Free book: Yet Again - R + Data Science

• Free book: Beautiful tables with {gt}

• Blog posts

Guten Tag!

Many greetings from Munich, Germany. Have you heard about brand.yml? It’s one of Posit’s newest and most exciting additions to their product lineup. With brand.yml, you can define a unified theme that works seamlessly across

• Quarto documents,

• Shiny for R apps, and

• Shiny for Python apps.

In today’s newsletter, I’ll walk you through what brand.yml is and how to use it to bring consistent styling to all your projects.

What’s the idea?

The idea behind brand.yml is simple but powerful:

1. You define your primary color, secondary color, fonts, logo, and other branding settings in a single brand.yml file.

2. You can then reuse that same configuration across all your data products. No need to redefine styles in each individual project.

This makes it easier to

• maintain consistent branding,

• reuse themes across projects, and

• share themes across teams.

How does this work?

You’re probably now wondering: What does a brand.yml file actually look like? Well, let me show you. It’s just a plain text YAML file. Here’s an example from the official docs:

As you can see, it looks pretty simple. But this simple file becomes the central definition of your app or document’s look and feel.

How to Use brand.yml?

The way you use this file depends on your toolchain. Here’s a breakdown for each use case:

📄 In Quarto

First, you need to save the branding content into a file called _brand.yml. And to apply the brand in a Quarto document you just have to add the brand attribute to the YAML header and fill it with the path to your yml-file. This could look something like this:

It doesn’t get any easier than that. And if you want to use the specified branding options (e.g. colors) in a particular setting (like Typst PDFs), then you can check out the dedicated Quarto documentation on that.

💻 In Shiny for R

In Shiny for R, brand.yml is integrated via the fantastic {bslib} package.
You can use it in your UI definition like so:

🐍 In Shiny for Python

You’ll need the latest version of Py-Shiny installed with the theme extra.

Once you’ve got that, you can use brand.yml like this:

And if you’re using Py-Shiny Express, the usage is slightly different. There, you’ll have to configure the page_options argument in the UI object.

I think brand.yml is a fantastic idea. It simplifies theming, supports consistency, and works beautifully across multiple frameworks. If you’re building out a data product pipeline that includes reports and interactive apps, this tool will save you time and help keep everything looking polished and on-brand.

Let me know what you think about brand.yml. And as always, if you have any questions, or just want to reach out, feel free to contact me by replying to this mail or finding me on LinkedIn or on Bluesky.

See you next week,
Albert 👋

Enjoyed this newsletter? Here are other ways I can help you:

Data Cleaning With R Master Class

A data scientist’s guide to avoid wasting time with data cleaning. Learn 5 fundamental aspects of data cleaning, get to your insights much quicker.

data-cleaning.albert-rapp.de

Insightful Data Visualizations for "Uncreative" R Users

In this course, I teach you how to create insightful data visualizations without being a design expert.

arapp.thinkific.com/courses/insightful-data-visualizations-for-uncreative-r-users

Making Beautiful Tables with R

Tables are often boring, hard to read, and, most importantly, they don't communicate effectively. In this course, I will show you that creating beautiful tables is just a matter of learning a few important principles.

rfortherestofus.com/courses/tables

• My playlist on YouTube

• Free book: Yet Again - R + Data Science

• Free book: Beautiful tables with {gt}

• Blog posts

Guten Tag!

Many greetings from Munich, Germany. Recently, I’ve been playing around with the new Positron IDE, and I must say: I really enjoy it. It still has some small bugs but overall it’s great. More importantly, it’s packed with features tailored for data science workflows.

In this week’s newsletter, I want to share my favorite features, settings, and shortcuts in Positron. That way, you can get a sense of why it might be a better fit than VSCode for your R or Python work.

Fixed Console for R and Python

One of the standout features in Positron is the fixed console for R and Python that never goes away. Unlike VSCode, there’s always a R or Python console that you can use for quick iterations. Also, you can easily switch between languages and even between different versions of R and Python.

And regardless of what you use, there’s a dedicated console that always receives your commands when you press Ctrl + Enter in the editor. This makes it far more reliable than what I experienced with VS Code, where the console sometimes wouldn’t respond or would detach from the current session.

And to make it perfectly clear, just for this feature alone, I’d already recommend Positron over VS Code.

Integrated Help Pane (F1 to View Function Docs)

Just like in RStudio, Positron lets you press F1 on any function in your script to open the Help page.

No more switching to a browser or dealing with limited pop-ups like in VS Code. This feature is built-in and persistent. It’s especially handy when you have to refer to arguments or function references on the fly.

The only issue that I have is that the formatting in the help windows isn’t always great. In particular, if you increase your editor font size. In that case, you will have a large code font and regular text font inside the help panel.

That’s not particularly great but I hope the kind people behind Positron are already on it. For now a workaround is to increase the window size instead if you want to have larger fonts.

Side-by-Side Layout (Just Like RStudio)

I love Positron’s layout customization. When you download the newest version, there’s a button in the top-right corner to switch layouts.

I always go with the side-by-side layout, similar to what I used in RStudio:

• Left: Script or Quarto file

• Right: Console, plots, and environment

This makes transitioning from my previous RStudio setup feel completely natural.

Zen Mode for Maximum Focus

Positron has a fantastic Zen mode that lets you minimize distractions and maximize space. That way, only the essential UI things are maximized. Here’s how it looks as I’m writing this newsletter.

Depending on what I need at the moment, I can toggle between:

• Code only

• Code + Console

• Code + Console + File Explorer

Familiar Shortcuts from RStudio

Positron supports many of your favorite RStudio shortcuts:

• Ctrl + Shift + M → Insert pipe (|>)

• Ctrl + Shift + K → Render Quarto document

I’ve also customized a few additional shortcuts to toggle the sidebars for Zen mode (like I’ve just described). Or to jump between console and editor.

And in case this feels intimidating, there’s also a UI that you can use to tweak these things. It’s just easier to share the settings with you this way.

Use the Command Palette (Ctrl + Shift + P)

The Command Palette is your best friend in Positron. Press Ctrl + Shift + P to access any command or setting.

Want to enter Zen Mode? Type “Zen” and the relevant action will pop up. You’ll also see the keyboard shortcut for it, helping you memorize it over time

Pro tip: If you delete the arrow > that shows up in the palette, it becomes a fuzzy finder for files and objects in your workspace.

Projects? Nope. Directories Instead.

Positron doesn’t use “projects” the way RStudio does. Instead, it uses directories. If you had an RStudio project before, just open the directory in Positron using: Ctrl + K followed by Ctrl + O 

That directory becomes your workspace root. And everything inside it is visible in the file explorer. Pretty much like RStudio projects.

Customize Everything with settings.json

Like VSCode, Positron is fully customizable via the settings.json file. Open it via the Command Palette and throw in any setting you want. Here are some of my favorites:

And just like with the keybindings.json, there’s also a UI that you can use.

Positron is a joy to use. It brings the best of VSCode’s extensibility and combines it with RStudio’s comfort, especially for data scientists. Let me know if you’re using it already and what your favorite tweaks are. I’d love to hear what’s working for you!

As always, if you have any questions, or just want to reach out, feel free to contact me by replying to this mail or finding me on LinkedIn or on Bluesky.

See you next week,
Albert 👋

Enjoyed this newsletter? Here are other ways I can help you:

Data Cleaning With R Master Class

A data scientist’s guide to avoid wasting time with data cleaning. Learn 5 fundamental aspects of data cleaning, get to your insights much quicker.

data-cleaning.albert-rapp.de

Insightful Data Visualizations for "Uncreative" R Users

In this course, I teach you how to create insightful data visualizations without being a design expert.

arapp.thinkific.com/courses/insightful-data-visualizations-for-uncreative-r-users

Making Beautiful Tables with R

Tables are often boring, hard to read, and, most importantly, they don't communicate effectively. In this course, I will show you that creating beautiful tables is just a matter of learning a few important principles.

rfortherestofus.com/courses/tables

• My playlist on YouTube

• Free book: Yet Again - R + Data Science

• Free book: Beautiful tables with {gt}

• Blog posts

Guten Tag!

Many greetings from Munich, Germany. In today’s newsletter, I want to show you how you can combine the {pins} package with GitHub Actions. That way, you can automate the process of downloading data from the internet and saving it as a time series.

The workflow is pretty similar to one of our previous newsletter on GitHub Actions. So if you’ve read that newsletter, then you should have no problem following along.

A repo for our project

The first thing we need is a new project for our data extraction endeavor. But let’s not start completely from scratch. Let’s use our “gh_actions_test” repo. It already has an {renv} environment for version control set up. That way, we don’t have to worry about that.

Add pins to {renv}

Now we can add {pins} to our {renv} environment. Simply call the install() command.

Then, we can create a new R-Script pins-demo.R in which we’ll actually use {pins}. For now, this should only contain this code:

That way, {renv} lets us write the {pins} package to its lock file. If you recall, this is the thing we have to restore at the beginning of every GH workflow. Now make sure that

• you can execute this code in your project, i.e. you have AWS credentials set,

• that you have your .Rprofile file removed from the git repo and added it to .gitignore (because you don’t want to share your keys),

• and that you saved the packages to the lock file via renv::snapshot().

Alright, cool. So with that we need a new GH actions workflow file.

Base GitHub Actions Workflow

Let’s start out with .yml workflow file from our last GitHub Actions tutorial. That’s a good place to start I’d say.

In this project, we don’t need anything from Quarto and we also don’t need to commit to Github, so we can remove those steps.

Unfortunately, R isn’t contained in the ubuntu-latest machine from GitHub anymore. That’s why we have to add R ourselves. But that’s super easy to do. We only have to use one of the R containers provided from the Rocker Project.

And with that we are once again ready to use R automations.

Extract Data

Next, we need an R function that fetches data from an external source. This could be via

• API calls,

• web scrapings, or

• any other data retrieval method.

To keep things simple, we’ll simulate it with a function that just returns a random value:

Check if pin is available

In the next step, we want to write this data to a pin that is stored in our board in S3. For that, we first need to check if our pin is even available. If not, we simply write to that pin without doing anything else.

Grab previous data

If the pin is already available, then we should

• grab the already saved data,

• append our new data to it, and

• write that new complete data to the same pin.

And that’s it. {pins} will do all the data versioning for us. So overall, our pins-demo.R script is only a short little file like this:

Modify GH workflow

Sweet. Now, we only have to tell our GH Actions workflow to run that script. The easiest way is to add another step after all dependencies are installed.

Add AWS Credentials to GitHub Secrets

But if you think about it, this cannot work yet. The machine that is supposed to execute this code, doesn’t know our GitHub credentials. If this script works on the GH action, then you should seriously check if you’ve accidentally pushed your .Rprofile file (which you should absolutely not do.)

So that’s why we have to tell our step about the environment variables it needs.

And then in your GitHub repository, go to Settings > Secrets and variables > Actions, and add:

• AWS_ACCESS_KEY_ID

• AWS_SECRET_ACCESS_KEY

• AWS_DEFAULT_REGION

And that’s it! Now every time the flow runs, we will have added a new row to our dataset.

You now have a working automation that pulls data from anywhere you like and stores it in cloud storage securely and automatically.

As always, if you have any questions, or just want to reach out, feel free to contact me by replying to this mail or finding me on LinkedIn or on Bluesky.

See you next week,
Albert 👋

Enjoyed this newsletter? Here are other ways I can help you:

Data Cleaning With R Master Class

A data scientist’s guide to avoid wasting time with data cleaning. Learn 5 fundamental aspects of data cleaning, get to your insights much quicker.

Insightful Data Visualizations for "Uncreative" R Users

In this course, I teach you how to create insightful data visualizations without being a design expert.

Making Beautiful Tables with R

Tables are often boring, hard to read, and, most importantly, they don't communicate effectively. In this course, I will show you that creating beautiful tables is just a matter of learning a few important principles.

• My playlist on YouTube

• Free book: Yet Again - R + Data Science

• Free book: Beautiful tables with {gt}

• Blog posts

Guten Tag!

Many greetings from Munich, Germany. In this week’s newsletter, I’ll show you how to use the {pins} package to store your data on different online platforms. Basically, this is a follow-up to last week’s newsletter.

And there are many places where you can store pins. Here, we’ll walk through two particuclar examples. Namely, GitHub and AWS S3. But before I show you how they work, let me show you my usual announcements.

Interactive Documents > Full App

Ever found yourself building a Shiny app that contains mostly text with just a few interactive parts? That’s a bit of a nuisance to set up because you’re mostly trying to make your text look nice inside your app’s big blank canvas.

Now here’s a better way.

Just create a Quarto document! This will immediately use formatting that is proper for text-heavy documents.

And the interactive part?

Well, in Quarto you can add them via Shiny server code chunks. And in my newest YT video I show you how that works. Check it out:

Using GitHub as a Pin Board

Let’s assume you have a public GitHub repository with pins stored in it. You can connect to it using the board_url() function. Here’s an example of that using pins from the public {pins} repo.

Once you’ve created the board, you can read pins from it using the pin_read() function:

But this only works with pins that contain actual R data sets. For example the raw pin contains a .txt file. That’s why it can’t be opened in the R console.

Instead, you have to download the file.

Using GitHub with Private Repos

What if you want to use a private repository instead?

In that case, you’ll need to include your GitHub Personal Access Token in the request. You do this by first creating a named vector with your token as the authorization header.

And then you could pass it to the headers argument of board_url(). Here, I’ve just taken the example from the {pins} docs.

Limitations of Pins with GitHub

Note that using pins like that only allows reading from GitHub. It doesn’t let you modify the data. If you want to change pins from a GitHub repo, you’d have to:

1. Clone the repository

2. Modify the pins locally

3. Push the changes back to GitHub

This isn’t the most convenient workflow. That’s why I prefer other options like AWS S3.

Using AWS S3 as a Pin Board

For a smoother experience, AWS S3 is a great option.
To use it, you need an existing S3 bucket to store your pins. If you don’t have one yet, check out the tutorial I wrote a few weeks back on how to create an S3 bucket.

Once you have your bucket and region ready, you can connect using the board_s3() function:

If you encounter an “Access Denied” error, it’s likely because your AWS credentials are not set up correctly. You need to have the following environment variables configured:

• AWS_ACCESS_KEY_ID

• AWS_SECRET_ACCESS_KEY

You can set these in your .Renviron file or in your R session.

Writing Pins to AWS S3

Once your S3 board is set up and authenticated, you can read and write pins just like with a local board.

The key benefit here is that you don’t need to manually download or push data. {pins} handles it all automatically. In fact, if you take a look at your S3 bucket now, you’ll see that the data is already there.

With that, you have two solid ways to store your pins online. Next week, I’ll show you how to set up an automation on GitHub Actions that automatically collects time series data and stores it using the {pins} package.

As always, if you have any questions, or just want to reach out, feel free to contact me by replying to this mail or finding me on LinkedIn or on Bluesky.

See you next week,
Albert 👋

Enjoyed this newsletter? Here are other ways I can help you:

Data Cleaning With R Master Class

A data scientist’s guide to avoid wasting time with data cleaning. Learn 5 fundamental aspects of data cleaning, get to your insights much quicker.

data-cleaning.albert-rapp.de

Insightful Data Visualizations for "Uncreative" R Users

In this course, I teach you how to create insightful data visualizations without being a design expert.

arapp.thinkific.com/courses/insightful-data-visualizations-for-uncreative-r-users

Making Beautiful Tables with R

Tables are often boring, hard to read, and, most importantly, they don't communicate effectively. In this course, I will show you that creating beautiful tables is just a matter of learning a few important principles.

rfortherestofus.com/courses/tables

• My playlist on YouTube

• Free book: Yet Again - R + Data Science

• Free book: Beautiful tables with {gt}

• Blog posts

Guten Tag!

Many greetings from Munich, Germany. In today’s newsletter, we are talking about the {pins} package. This package is not very well known in the R sphere, but it is a fantastic tool for keeping track of different data versions.

Basically, {pins} makes storing data and versioning dead-simple. You can think of it as Git for your data. But before we dive in, I have a short thing to ask you: ‘

What‘s next?

Recently, I find myself covering more and more technical aspects of data science that are not necessarily R focused. This includes:

• Building automations with GitHub Actions

• Setting up infrastructure on AWS or Azure

• Using JavaScript to handle AI responses in Shiny apps

I do this because I think data science doesn’t happen in any single tool or language. Thus, I want to continue writing in as broad a scope as I feel like at the moment (after all I get to do whatever I want on this newsletter 🤪)

But I do want to see if I can make this as valuable for you as possible. So let me know what you’re interested in by filling out this small poll.

Just click on your favorite choice and it will be recorded. After that, you’ll also have the chance to leave additional comments in case you want to suggest something I haven’t thought of.

Why use the {pins} package?

Now, before I show you how {pins} works, let’s talk about why you should even use it

• Version control: Prevents accidental overwriting of important datasets when re-running workflows.

• Efficient workflows: You can run a calculation or an API call once and store the data permanently in a pin.

• Metadata storage: {pins} keeps detailed information about your data so you can easily restore or revert to previous versions.

• Seamless integration : You can store data anywhere you like. GitHub, Google Drive, AWS S3, you name it. {pins} will work exactly the same.

Surely, you’re wondering how all of that is supposed to work. So let’s dive into the technical details.

Set up {pins}

First, you need to install the package. As always, you can do this with:

Set up a board

To store data, you need to place “pins” in a “board”. If you want to try {pins} locally first, create a board using:

Write data to a pin

To write data to your board, simply use:

This creates a new pin, and the best part is that previous data versions aren’t deleted.

Retrieving a specific version

To check how many data versions exist in your board, use:

That’s currently pretty boring. So let’s modify our data and overwrite our pin multiple times.

Now, check out the different versions:

Once you decide which version you want to load, retrieve it using:

As you can see, setting up {pins} is simple and efficient for storing and retrieving data. And the best thing is that you cannot accidentally overwrite data. Next week, I’ll show you how to set up cloud storages for your pins.

As always, if you have any questions, or just want to reach out, feel free to contact me by replying to this mail or finding me on LinkedIn or on Bluesky.

See you next week,
Albert 👋

Enjoyed this newsletter? Here are other ways I can help you:

Data Cleaning With R Master Class

A data scientist’s guide to avoid wasting time with data cleaning. Learn 5 fundamental aspects of data cleaning, get to your insights much quicker.

Insightful Data Visualizations for "Uncreative" R Users

In this course, I teach you how to create insightful data visualizations without being a design expert.

Making Beautiful Tables with R

Tables are often boring, hard to read, and, most importantly, they don't communicate effectively. In this course, I will show you that creating beautiful tables is just a matter of learning a few important principles.

• My playlist on YouTube

• Free book: Yet Again - R + Data Science

• Free book: Beautiful tables with {gt}

• Blog posts