Outils pour utilisateurs

Outils du site


php:magnyaa

Magnyaa


Status: 1.0 “The Internet Is Made Of Cats, Nyaa~ !” released; pausing

Jump to downloads


[ Description ]

Magnyaa starts where Magnesia ended.

The focus remains the same: provide a lightweight database mirror for NyaaTorrents (nyaa.eu), based on BitTorrent magnet links. Now the project is divided in 2 parts: the engine and the browser.

Magnyaa Engine

The Magnyaa Engine is written in PHP5 and works with SQLite3 databases. Each database contains up to 1000 torrent informations and weights ~130 kB (3 times more with complete updated data). The Engine responds with JSON messages. The Engine functions are:

  1. provide specific info for 1 torrent (based on Nyaaid)
  2. provide info for 1 full database (based on database number)
  3. let admin remove specific info for 1 torrent (based on Nyaaid)
  4. let admin insert specific info for 1 torrent (based on Nyaaid)
  5. push all databases of all instances in zip file and export it
  6. get the newest nyaaid of the newest database of all instances
  7. check the existence of a given database

Magnyaa Engine is actually designed with 2 instances: Nyaa's main website and Nyaa's “sukebei” website.

The JSON response messages allows using Magnyaa Engine with many applications and languages, may it be PHP, Javascript, Python …

Engine Feeder

The Engine Feeder consists of 3 bash scripts and one PHP script (which is torrent-rw). One bash script for each instance, plus a “cron-job script” to ease execution of both and avoid overlapping (since they use the same PHP resources). So you need a GNU/Linux machine with basic tools (wget, awk, head, sed,..), php5-cli and httping. Well actually you can replace httping with “wget –spider”. But everything you need can be found on Debian, Ubuntu, Mandriva, Archlinux, Fedora,..

Windows people can use virtual machines (like vmware or virtualbox), it just needs internet access.

How they work: they get the newest known torrents from both Nyaa and the Engine, compare them, and if Engine is missing some they download the details page and the torrent itself, parse data like the addition date and torrent name from the first, size and BTIH from the second. The data is then formatted in CSV, base64-encoded and sent to the Engine.

Cron jobs can be set at any time, I recommend around 15 minutes for fast adding, but every hour is ok. Lockfiles are created and checked so there won't be any cronjob overcrowding when one takes longer than usual.

Magnyaa Browser

The Magnyaa Browser is written in PHP5 and parses JSON messages.

Which of the “main” and “sukebei” instances is used is set in the config.php. Well actually the config.php defines quite much: name, Engine URL, export link display, links between instances, disclaimer text, headers, CSS, footer, Engine check, category code translation,.. Use a real text editor so you can easily find and read the comments :) Windows' default Notepad won't make it, absolutely not (prefer Notepad++)

The provided download of Magnyaa Browser is pre-configured with the project's test Engine: just upload it and you're ready to go. But keep one thing in mind: it's a test Engine, and retro-compatibility isn't our priority. We may have designed the Engine so that it won't change much in the way of responding to requests, but keep an eye on updates: if your Magnyaa Browser suddenly shows “error with Engine.. is it one ?”, update it.

Of course you can use your own Engine and feed it yourself, so you have a really independent mirror.


Open source and relying on widely available technologies like PHP5, SQLite3, JSON, HTML5, and CSS3, Magnyaa is one of my most ambitious personal development projects. Sorry for the messy code, hopefully it still will inspire people to build upon this :-)

[ Download ]

[ Installing and Updating ]

Browser

  • upload
  • (customize config.php at will)
  • done !
  • update: replace index.php, compare and edit config.php, add any other file

Engine

  • RTFM. (it's included in the zip)

[ Changelog ]

Release 1 “The Internet Is Made Of Cats, Nyaa~ !”

  • (initial)

[ Engine technical details ]

  • index.php
  • config.php

→ data_#/{4 digits}.db

index.php: main worker

config.php: common parameters (like error reporting policy, SQLite3 timeout, admin password and mail)

data_#/{4 digits}.db: contains torrent data, up to 1000 elements

As for now, # is “m” for main and “s” for sukebei. Data is stored and fetched in databases named like: torrent id divided by 1000; padded to 4 digits to the left with zeros. For example, “351540” would be in database “0351.db”. This provides space for over 9 million torrent informations, more than a century time at Nyaa's actual pace ;-)


Queries:

Note: if {id} is starting with “s”, the considered instance is “sukebei”. Else, “main”.

no query: empty page (actually useful: the HTTP header “X-Magnyaa: engineX” that tells Browser it's an Engine and it is of version X)

?nyaaid={id}: check if existing database, load it, SELECT the {id}, parse data, output in JSON

Note: if “updated” is older than 1 hour (or NULL), the Engine fetches and parses the torrent details from Nyaa, and updates the columns with the data (description, information, seeders, leechers, downloads, comments, updated). Seeders, leechers, downloads and updated are raw numbers, the rest is base64_encoded before UPDATE.

In case the torrent doesn't seem to be on Nyaa anymore, the Engine automatically REMOVE the torrent on request.

?db={id}: check if existing database, load it, SELECT all, parse data, output in JSON

?remove={id}&p={pass}: DELETE FROM database according to {id}

?import={string}&p={pass}: base64_decode {string}, parse the CSV, DELETE {id} (in case it already exists), INSERT the parsed data.

?export: if export.zip not expired, redirect to it. Else create ziparchive of data_#/ as export.zip, and redirect to it.

?latest: SELECT the newest torrent of the newest database of all instances, output it's nyaaid in JSON

?exist={id}: check if database {id} exists. Output result in JSON.

Magnyaa Engine and Browser are published under the GNU GPLv3 license.

The Engine Feeder uses torrent-rw, published under GNU GPLv3 by Adrien Gibrat: https://github.com/adriengibrat/torrent-rw/

Magnyaa in itself is neutral and automated, the developers can't be held liable for any misuse of the software. If Nyaa.eu's team made mistakes, well.. blame them. Magnyaa is a “set up and forget” thing.

[ Contact ]

Bug report, feature requests, anything else ? Head over here:

http://www.suumitsu.eu/contact (don't mind the french stuff, just proceed)

php/magnyaa.txt · Dernière modification : 2014-09-07 13:14 de mitsu