homebr
- Arch:
yay -S archivebox (contributed by @imlonghao)
-- FreeBSD:
curl -sSL 'https://get.archivebox.io' | sh (uses pkg + pip3 under-the-hood)
+- FreeBSD:
curl -fsSL 'https://get.archivebox.io' | sh (uses pkg + pip3 under-the-hood)
- Nix:
nix-env --install archivebox (contributed by @siraben)
- Guix:
guix install archivebox (contributed by @rakino)
- More: contribute another distribution...!
@@ -458,14 +467,15 @@ ArchiveBox commands can be run in a terminal directly on your host, or via Docke
mkdir -p ~/archivebox/data # create a new data dir anywhere
cd ~/archivebox/data # IMPORTANT: cd into the directory
-# archivebox [subcommand] [--args]
+# archivebox [subcommand] [--help]
+archivebox version
archivebox help
-# equivalent: docker compose run archivebox [subcommand [--args]
+# equivalent: docker compose run archivebox [subcommand] [--help]
docker compose run archivebox help
-# equivalent: docker run -it -v $PWD:/data archivebox/archivebox [subcommand [--args]
- docker run -it -v $PWD:/data archivebox/archivebox help
+# equivalent: docker run -it -v $PWD:/data archivebox/archivebox [subcommand] [--help]
+docker run -it -v $PWD:/data archivebox/archivebox help
```
#### ArchiveBox Subcommands
@@ -482,7 +492,7 @@ docker compose run archivebox help
# make sure you have pip-installed ArchiveBox and it's available in your $PATH first
-# archivebox [subcommand] [--args]
+# archivebox [subcommand] [--help]
archivebox init --setup # safe to run init multiple times (also how you update versions)
archivebox version # get archivebox version info + check dependencies
archivebox help # get list of archivebox subcommands that can be run
@@ -498,7 +508,7 @@ archivebox add --depth=1 'https://news.ycombinator.com'
# make sure you have `docker-compose.yml` from the Quickstart instructions first
-# docker compose run archivebox [subcommand [--args]
+# docker compose run archivebox [subcommand] [--help]
docker compose run archivebox init --setup
docker compose run archivebox version
docker compose run archivebox help
@@ -515,7 +525,7 @@ docker compose run archivebox add --depth=1 'https://news.ycombinator.com'
# make sure you create and cd into in a new empty directory first
-# docker run -it -v $PWD:/data archivebox/archivebox [subcommand [--args]
+# docker run -it -v $PWD:/data archivebox/archivebox [subcommand] [--help]
docker run -v $PWD:/data -it archivebox/archivebox init --setup
docker run -v $PWD:/data -it archivebox/archivebox version
docker run -v $PWD:/data -it archivebox/archivebox help
@@ -675,7 +685,7 @@ It uses all available methods out-of-the-box, but you can disable extractors and
Expand to see the full list of ways it saves each page...
-./archive/{Snapshot.id}/
+data/archive/{Snapshot.id}/
- Index:
index.html & index.json HTML and JSON index files containing metadata and details
- Title, Favicon, Headers Response headers, site favicon, and parsed site title
@@ -806,18 +816,18 @@ All of ArchiveBox's state (SQLite DB, content, config, logs, etc.) is stored in
Expand to learn more about the layout of Archivebox's data on-disk...
-Data folders can be created anywhere (`~/archivebox` or `$PWD/data` as seen in our examples), and you can create as many data folders as you want to hold different collections.
+Data folders can be created anywhere (`~/archivebox/data` or `$PWD/data` as seen in our examples), and you can create as many data folders as you want to hold different collections.
All archivebox CLI commands are designed to be run from inside an ArchiveBox data folder, starting with archivebox init to initialize a new collection inside an empty directory.
-mkdir ~/archivebox && cd ~/archivebox # just an example, can be anywhere
+mkdir -p ~/archivebox/data && cd ~/archivebox/data # just an example, can be anywhere
archivebox init
-The on-disk layout is optimized to be easy to browse by hand and durable long-term. The main index is a standard index.sqlite3 database in the root of the data folder (it can also be exported as static JSON/HTML), and the archive snapshots are organized by date-added timestamp in the ./archive/ subfolder.
+The on-disk layout is optimized to be easy to browse by hand and durable long-term. The main index is a standard index.sqlite3 database in the root of the data folder (it can also be exported as static JSON/HTML), and the archive snapshots are organized by date-added timestamp in the data/archive/ subfolder.
-/data/
+data/
index.sqlite3
ArchiveBox.conf
archive/
@@ -832,7 +842,7 @@ The on-disk layout is optimized to be easy to browse by hand and durable long-te
...
-Each snapshot subfolder ./archive/TIMESTAMP/ includes a static index.json and index.html describing its contents, and the snapshot extractor outputs are plain files within the folder.
+Each snapshot subfolder data/archive/TIMESTAMP/ includes a static index.json and index.html describing its contents, and the snapshot extractor outputs are plain files within the folder.
Learn More
@@ -1046,9 +1056,9 @@ Because ArchiveBox is designed to ingest a large volume of URLs with multiple co
Disk usage can be reduced by using a compressed/deduplicated filesystem like ZFS/BTRFS, or by turning off extractors methods you don't need. You can also deduplicate content with a tool like [fdupes](https://github.com/adrianlopezroche/fdupes) or [rdfind](https://github.com/pauldreik/rdfind).
-**Don't store large collections on older filesystems like EXT3/FAT** as they may not be able to handle more than 50k directory entries in the `archive/` folder.
+**Don't store large collections on older filesystems like EXT3/FAT** as they may not be able to handle more than 50k directory entries in the `data/archive/` folder.
-**Try to keep the `index.sqlite3` file on local drive (not a network mount)** or SSD for maximum performance, however the `archive/` folder can be on a network mount or slower HDD.
+**Try to keep the `data/index.sqlite3` file on local drive (not a network mount)** or SSD for maximum performance, however the `data/archive/` folder can be on a network mount or slower HDD.
If using Docker or NFS/SMB/FUSE for the `data/archive/` folder, you may need to set [`PUID` & `PGID`](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#puid--pgid) and [disable `root_squash`](https://github.com/ArchiveBox/ArchiveBox/issues/1304) on your fileshare server.
@@ -1439,7 +1449,7 @@ https://stackoverflow.com/questions/1074212/how-can-i-see-the-raw-sql-queries-dj
ArchiveBox [`extractors`](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/extractors/media.py) are external binaries or Python/Node scripts that ArchiveBox runs to archive content on a page.
-Extractors take the URL of a page to archive, write their output to the filesystem `archive/TIMESTAMP/EXTRACTOR/...`, and return an [`ArchiveResult`](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/models.py#:~:text=return%20qs-,class%20ArchiveResult,-(models.Model)%3A) entry which is saved to the database (visible on the `Log` page in the UI).
+Extractors take the URL of a page to archive, write their output to the filesystem `data/archive/TIMESTAMP/EXTRACTOR/...`, and return an [`ArchiveResult`](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/models.py#:~:text=return%20qs-,class%20ArchiveResult,-(models.Model)%3A) entry which is saved to the database (visible on the `Log` page in the UI).
*Check out how we added **[`archivebox/extractors/singlefile.py`](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/extractors/singlefile.py)** as an example of the process: [Issue #399](https://github.com/ArchiveBox/ArchiveBox/issues/399) + [PR #403](https://github.com/ArchiveBox/ArchiveBox/pull/403).*
diff --git a/archivebox/config.py b/archivebox/config.py
index be75ffdc..3da38b3f 100644
--- a/archivebox/config.py
+++ b/archivebox/config.py
@@ -154,7 +154,7 @@ CONFIG_SCHEMA: Dict[str, ConfigDefaultDict] = {
'CHROME_SANDBOX': {'type': bool, 'default': lambda c: not c['IN_DOCKER']},
'YOUTUBEDL_ARGS': {'type': list, 'default': lambda c: [
'--restrict-filenames',
- '--trim-filenames',
+ '--trim-filenames', '128',
'--write-description',
'--write-info-json',
'--write-annotations',
@@ -366,24 +366,32 @@ ALLOWDENYLIST_REGEX_FLAGS: int = re.IGNORECASE | re.UNICODE | re.MULTILINE
############################## Version Config ##################################
-def get_system_user():
- SYSTEM_USER = getpass.getuser() or os.getlogin()
+def get_system_user() -> str:
+ # some host OS's are unable to provide a username (k3s, Windows), making this complicated
+ # uid 999 is especially problematic and breaks many attempts
+ SYSTEM_USER = None
+ FALLBACK_USER_PLACHOLDER = f'user_{os.getuid()}'
+
+ # Option 1
try:
import pwd
- return pwd.getpwuid(os.geteuid()).pw_name or SYSTEM_USER
- except KeyError:
- # Process' UID might not map to a user in cases such as running the Docker image
- # (where `archivebox` is 999) as a different UID.
- pass
- except ModuleNotFoundError:
- # pwd doesn't exist on windows
- pass
- except Exception:
- # this should never happen, uncomment to debug
- # raise
+ SYSTEM_USER = SYSTEM_USER or pwd.getpwuid(os.geteuid()).pw_name
+ except (ModuleNotFoundError, Exception):
pass
- return SYSTEM_USER
+ # Option 2
+ try:
+ SYSTEM_USER = SYSTEM_USER or getpass.getuser()
+ except Exception:
+ pass
+
+ # Option 3
+ try:
+ SYSTEM_USER = SYSTEM_USER or os.getlogin()
+ except Exception:
+ pass
+
+ return SYSTEM_USER or FALLBACK_USER_PLACHOLDER
def get_version(config):
try:
diff --git a/archivebox/system.py b/archivebox/system.py
index d80a2cb5..6e03846f 100644
--- a/archivebox/system.py
+++ b/archivebox/system.py
@@ -146,20 +146,24 @@ def get_dir_size(path: Union[str, Path], recursive: bool=True, pattern: Optional
recursively and limiting to a given filter list
"""
num_bytes, num_dirs, num_files = 0, 0, 0
- for entry in os.scandir(path):
- if (pattern is not None) and (pattern not in entry.path):
- continue
- if entry.is_dir(follow_symlinks=False):
- if not recursive:
+ try:
+ for entry in os.scandir(path):
+ if (pattern is not None) and (pattern not in entry.path):
continue
- num_dirs += 1
- bytes_inside, dirs_inside, files_inside = get_dir_size(entry.path)
- num_bytes += bytes_inside
- num_dirs += dirs_inside
- num_files += files_inside
- else:
- num_bytes += entry.stat(follow_symlinks=False).st_size
- num_files += 1
+ if entry.is_dir(follow_symlinks=False):
+ if not recursive:
+ continue
+ num_dirs += 1
+ bytes_inside, dirs_inside, files_inside = get_dir_size(entry.path)
+ num_bytes += bytes_inside
+ num_dirs += dirs_inside
+ num_files += files_inside
+ else:
+ num_bytes += entry.stat(follow_symlinks=False).st_size
+ num_files += 1
+ except OSError:
+ # e.g. FileNameTooLong or other error while trying to read dir
+ pass
return num_bytes, num_dirs, num_files
@@ -25,7 +25,6 @@ curl -sSL 'https://get.archivebox.io' | sh # (or see pip/brew/Docker instruct
Without active preservation effort, everything on the internet eventually dissapears or degrades. Archive.org does a great job as a centralized service, but saved URLs have to be public, and they can't save every type of content.
*ArchiveBox is an open source tool that lets organizations & individuals archive both public & private web content while retaining control over their data. It can be used to save copies of bookmarks, preserve evidence for legal cases, backup photos from FB/Insta/Flickr or media from YT/Soundcloud/etc., save research papers, and more...*
-