I started this project to get my feet wet using Go. I’ve been primarily a JavaScript/TypeScript developer throughout my career and have been wanting to dive deeper into other languages.

I will not bore you with all the details or syntax of Go as other websites are a proper place for that, but to share the initial steps I’ve done, I have read the first few chapters of this book Introducing Go: Build Reliable, Scalable Programs by Caleb Doxsey to get a feel on how you write programs in Go.

What and Why

Library Genesis is a file-sharing-based shadow library website for scholarly journal articles, academic and general-interest books, images, comics, audiobooks, and magazines (Wikipedia)

I have used this website for acquiring pdfs and other learning resources and materials. To be completely clear: I am not in any way condoning piracy or copyright infringement. If you want to support the authors I would recommend you to buy their books.

Motivation and Gist of the project

I have been using the website for around a year or two to quickly get my hands on different reading materials/publications but using it has always been a pain because of its terrible user interface and multiple-page redirection before you could download the file itself. I want to use the CLI tool to search and download the file(s) I need, my idea was something like clibgen search "Eloquent JavaScript" and would just to choose using the arrow keys on my keyboard which file I want.

Project Setup

I have set up my Go environment properly at this point

Initially, I played with the standard and built-in net/http package and io/ioutil to get an idea of how to send an HTTP GET request and read the response body. Coming from JS/TS world it was quite interesting to see that multiple return values are widely used in Golang, especially for error checking and error handling, although we have an object and array destructuring in JavaScript most of the code I touched doesn’t use it the way Go used multiple values. And I think they are built for an entirely different purpose, the syntax just looks a little bit similar.

Go has built-in support for multiple return values. This feature is used often in idiomatic Go, for example, to return both result and error values from a function. - gobyexample.com

In this code snippet below I did not handle the error that would be returned by http.Get() and ioutil.ReadAll() by using a blank identifier _.

package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
)

func main() {
	resp, _ := http.Get("https://libgen.is/search.php?req=Golang&lg_topic=libgen&open=0&view=simple&res=25&phrase=1&column=def")

	defer resp.Body.Close()
	body, _ := ioutil.ReadAll(resp.Body)

	sb := string(body)
	fmt.Println(sb)
}

This code snippet allows us to send an HTTP request and get the response body (HTML).

Type Definition

Type definitions in go are relatively straightforward and is very similar to TypeScript.

package API

type Book struct {
	ID        string
	Title     string
	Author    string
	Year      string
	Publisher string
	Extension string
	Mirrors   []string
	FileSize  string
}

The type Book here is pretty self-explanatory aside from the Mirrors. The Mirrors here are just the slices of string for multiple download link mirrors. (As of this writing I’m currently just using the first mirror, but plan to have a fallback or flexibility to use other mirrors)

An array has a fixed size. A slice, on the other hand, is a dynamically-sized, flexible view into the elements of an array. In practice, slices are much more common than arrays. - go.dev

Goquery

Since it would be a chore to manually parse the data we need from the page, I looked up a package that would help me read, parse and traverse an HTML document with ease. This is how I found Goquery which I guess is a play on its name to mimic jQuery.

The great thing about modern software development and having a mature ecosystem is that if you want to do something fast and you know exactly what you need there is probably an existing library for that out there. This is similar to how mature and most of the time bloated the JS ecosystem is.

GoQuery Github

Loading and reading the HTML document

According to the documentation creating a document is easy and supports an out-of-the-box response body from the HTTP library. So I do not need to read the stream and get the raw HTML response.

The following code looks similar to jQuery. We just pass the selector string to tell Goquery what we are looking for, in this case <tr>s that we need to get the data from each row. At this point, the code looks similar to JavaScript and callback functions.

document.Find(".c > tbody > tr").Each(func(resultsRow int, bookRow *goquery.Selection) {
  var id, author, title, publisher, extension, year, fileSize string
  var mirrors []string
  if resultsRow != 0 {
    bookRow.Find("td").Each(func(column int, columnSelection *goquery.Selection) {
	  switch column {
	    case 0:
		  id = columnSelection.Text()
		case 1:
		  author = columnSelection.Text()
		case 2:
		  title = getBookTitleFromSelection(columnSelection)
		case 3:
		  publisher = columnSelection.Text()
		case 4:
		  year = columnSelection.Text()
		case 7:
		  fileSize = columnSelection.Text()
		case 8:
		  extension = columnSelection.Text()
		case 9, 10, 11:
		  href, hrefExists := columnSelection.Find("a").Attr("href")
		  if hrefExists {
		    mirrors = append(mirrors, href)
		  }
		}
	  })
    books = append(books, Book{
      ID:        id,
	  Author:    author,
	  Year:      year,
	  Title:     title,
	  Publisher: publisher,
	  Extension: extension,
	  Mirrors:   mirrors,
	  FileSize:  fileSize,
    })
  }
})

Final Folder Structure

To give you an idea of what the final project folder structure would look like:

.
├── LICENSE
├── README.md
├── cmd
│   ├── root.go
│   └── search.go
├── go.mod
├── go.sum
├── main.go
└── pkg
    └── api
        ├── book.go
        ├── libgen.go

Look and Feel

Even though this is just a terminal application I would still like for it to be easy to use since I want to use this tool myself to download books quickly. I used promptui and cobra to improve the experience this gives me the ability to accept flags and have a nice UI for selecting which resources to download.

Final Notes

Overall it was a nice experience writing a working code in Go. But I have yet to learn the nitty-gritty parts of the language. The project setup was smooth, and documentation for the standard library is really straightforward. There are a lot of weird quirks that I have to get used to like an identifier is exported if it begins with a capital letter and so on.

Thanks for reading! Keep learning :)

Since I plan on continuously improving this tool, some code will not look 100% exactly the same on this repository and this article. Here’s the Github Link