Linux One Liner – Parsing long HTML urls

Ever wanted to look at a long HTML URL more easily, say to investigate a parameter. Here is a search from MapQuest.

http://www.mapquest.com/maps/map.adp?formtype=address&addtohistory=&address=10%20Market%20St&city=San%20Francisco&state=CA&zipcode=94111%2d4801&country=US&geodiff=1

<br /> $ echo "[insert url here]" | | tr "&#038;?" "n"<br />

This produced for the above URL the following output.
`

http://www.mapquest.com/maps/map.adp

formtype=address
addtohistory=
address=10%20Market%20St
city=San%20Francisco
state=CA
zipcode=94111%2d4801
country=US
geodiff=1
`

The Translate command tr does however replace both the & and ? characters. There are of course many more approaches like.

<br /> echo "[insert url here]" | sed -e "s/&#038;/\n/g" -e "s/?/\n/g"<br />

You can easily preserve the & and ? characters extending the syntax with
<br /> echo "[insert url here]" | sed -e "s/&#038;/\n&#038;/g" -e "s/?/\n?/g<br />

This produces.
`

http://www.mapquest.com/maps/map.adp

?formtype=address
&addtohistory=
&address=10%20Market%20St
&city=San%20Francisco
&state=CA
&zipcode=94111%2d4801
&country=US
&geodiff=1
`

Now don’t get me started with the awk command. One of my popular books is Sed & Awk. If you do any detailed Shell scripting, this is a very handy guide.

Tagged with: General Linux One Liners

Related Posts

Fixing Old Software Bugs Without Thinking

Fixing Old Software Bugs Without Thinking

I previously came across an issue in my benchmarking work where I relied on the Latency Histogram section of sysbench output. On Linux it worked fine — I generally ran my workload there, so it never mattered.

Read more
Curating DuckDB Datasets Leveraging AI

Curating DuckDB Datasets Leveraging AI

In Curated MySQL Data Sets for Realistic Testing I described datasets assembled the manual way — download, schema, load, validate, document — over hours or days per source. This post is the follow-up I promised: what changes when AI assists the same workflow, using DuckDB as the target engine and GeoNames as the first example.

Read more
Why INSERT IGNORE should not be used

Why INSERT IGNORE should not be used

Let’s say you’re building a reference table from source data, e.g. a silver medallion table from a primary source. In this example I am using a file of random locations on the globe extracted from OpenStreetMap (OSM) as my primary source.

Read more