Data
Curating DuckDB Datasets Leveraging AI
In Curated MySQL Data Sets for Realistic Testing I described datasets assembled the manual way — download, schema, load, validate, document — over hours or days per source. This post is the follow-up I promised: what changes when AI assists the same workflow, using DuckDB as the target engine and GeoNames as the first example.
Read more
Why INSERT IGNORE should not be used
Let’s say you’re building a reference table from source data, e.g. a silver medallion table from a primary source. In this example I am using a file of random locations on the globe extracted from OpenStreetMap (OSM) as my primary source.
Read more
What Constitutes Clean Data?
Clean data is one of those terms everyone uses and few define precisely. Row counts match? No nulls in key columns? Values within expected ranges? All useful checks — and all insufficient once you move from a table view to a visualization view.
Read more
Curated MySQL Data Sets for Realistic Testing
Synthetic benchmarks have their place, but I have always preferred working with real data. Not client production data — that stays private — but publicly available datasets that reflect the messy shapes, skewed distributions, and indexing challenges you encounter in the wild.
Read more
Creating a More Realistic Benchmark
Common benchmark approaches fall into two general categories, synthetic testing and realistic testing. You have the most generic operations from a synthetic test, starting with the most simple example using a single table, a single column, and for a single DML operation.
Read more
Your Attack Vector Extends Beyond Production Systems
A common data security issue is the unprotected copying of production data to non-production environments without any redaction, masking, or filtering. This practice poses a serious risk. A malicious actor will target the weakest link in your infrastructure, including non-production accounts and the developer systems accessing them.
Read more
Database testing for all version changes (including minor versions)
We know that SQL statement compatibility can change with major database version upgrades and that you should adequately test for them. But what about minor version upgrades? It is dangerous to assume that your existing SQL statements work with a minor update, especially when using an augmented version of an open-source database such as a cloud provider that may not be as transparent about all changes.
Read more
The curse of MySQL warnings
MySQL warnings are an anti-pattern when it comes to maintaining data integrity. When the information retrieved from a database does not match what was entered, and this is not identified immediately, this can be permanently lost.
Read more![Digital Tech Trek Digest [#Issue 2024.09]](https://ronaldbradford.com/images/blog/2024-digital-tech-trek-digest.png)
Digital Tech Trek Digest [#Issue 2024.09]
As an entrepreneur, pricing is an important consideration in any evaluation, development, and customer testing. In How To Price A SaaS Product , we see different pricing strategies, cost-based pricing, competitor-based pricing, penetration pricing, value-based pricing, freemium pricing.
Read moreDigital Tech Trek Digest [#Issue 2024.05]
Because the world needs better dashboards While my professional interests in Building Better Data Insights Faster rely on using visuals and narratives to show data-driven results, “Starting from first principles” is the question you have to ask.
Read moreData Masking 101
I continue to dig up and share this simple approach for production data masking via SQL to create testing data sets. Time to codify it into a post. Rather than generating a set of names and data from tools such as Mockaroo , it is more practical to use actual data for a variety of testing reasons.
Read moreA reliable and dependable application requires observability
Observability (o11y) is a critical pre-requisite component in software architecture when advocating for and preparing organizations for making informed decisions on the success of their application. Open Telemetry from the Cloud Native Computing Foundation is the goto standard regardless of your choices of monitoring tools.
Read moreOur Data Security Moonshot Starts With Prevention
The recent re-announcement of the Cancer Moonshot highlighted a common enemy to many endeavors to improve our society as a whole, and that is using common sense and already known methods.
Read moreCan a picture replace a text description?
Data visualization, data storytelling, and data lineage are all ways to better describe and visualize a specific situation for a set of data. Generally, I find these techniques are used as a means to uncover or identify information that ultimately pertains to individuals.
Read more