Manual for administrators

Hubzilla is more than a simple web application. It is a complex communication system that is more like an email server than a web server. For reliability and performance reasons, messages are delivered in the background and queued for later delivery when websites are down. This type of functionality requires a little more from the host system than the typical blog. Not every PHP/MySQL hosting provider is able to support Hubzilla. Many can, but please check the requirements and confirm with your hosting provider prior to installation. We have gone to great lengths to ensure that Hubzilla runs on standard hosting platforms such as those used for WordPress blogs and Drupal websites. It runs on most Linux systems.

Where to find further help

If you run into problems or have questions that are not covered in this documentation, please let us know via the Github Issue Tracker. Please describe your operating environment as accurately as possible and provide as much detail as possible about the error messages you see so that we can avoid them in the future. Due to the wide variety of operating systems and PHP platforms, we have limited ability to debug your PHP installation or obtain missing modules, but we will do our best to resolve common code issues.

Before you start

Select a domain name or a subdomain name for your server.

The software can only be installed in the root directory of a domain or subdomain and cannot be installed via alternative TCP ports. These restrictions may be relaxed in the future, but they are inconvenient, so we STRONGLY recommend that you continue to adhere to them.

Decide if you want to use SSL and obtain an SSL certificate before installing the software. You SHOULD use SSL. If you use SSL, you MUST use a ‘browser valid’ certificate. You CANNOT use self-signed certificates! Please test your certificate before installation. You can find a web tool for testing your certificate at ‘http://www.digicert.com/help/'. When you visit your website for the first time, please use the SSL URL (‘https://’) if SSL is available. This will avoid problems later on. The installation routine does not allow you to use a non-browser valid certificate.

This restriction is made because public posts by you may contain links to images in your own hub. Other members viewing your stream on other hubs will receive warnings if your certificate is not trusted by their web browser. This will confuse many people as it is a decentralised network and they will receive the warning about your hub while viewing their own hub and may think their own hub has a problem. These warnings are very technical and scary for some people, many of whom don't know how to proceed other than to follow the browser's advice. This is disruptive to the community. That said, we recognise the problems associated with the current certificate infrastructure and agree that there are many issues, but that doesn't change the requirement.

Free ‘browser valid’ certificates are available from providers such as ZeroSSL, LetsEncrypt and a few others. If you are NOT using SSL, there may be a delay of up to a minute on the first install script - while we check the SSL port to see if anything is responding there. When communicating with new sites, Hubzilla will always try to connect via the SSL port first before falling back to a less secure connection. If you are not using SSL, your web server does not HAVE to listen on port 443 at all.

If you are using LetsEncrypt to provide certificates and create a file under .well-known/acme-challenge so that LetsEncrypt can verify your domain ownership, please remove the .well-known directory or rename it once the certificate has been generated. The software provides its own handler for ‘.well-known’ services during installation, and an existing directory in this location may prevent some of these services from working correctly. This should not be a problem with Apache, but can be a problem with nginx or other web server platforms.

Installation

There are several ways to install a new hub.

• Manual installation on an existing server
• Automated installation on an existing server with a shell script
• Automated deployment via an OpenShift Virtual Private Server (VPS)
• Installation as Docker-Container

Automated installation via the shell script .homeinstall

There is a shell script in (.homeinstall/hubzilla-setup.sh) that will install Hubzilla and its dependencies on a fresh installation of Debian stable. It should work on similar Linux systems, but your results may vary.

Requirements

The installation script was originally developed for a small hardware server behind your home router. However, it has been tested on several systems running Debian 9:

• Home-PC (Debian-9.2-amd64) and Rapberry-Pi 3 (Rasbian = Debian 9.3)
  • Internet connection and router at home
  • Mini-PC / Raspi connected to the router
  • USB drive for backups
  • Fresh installation of Debian on your mini PC
  • Router with open ports 80 and 443 for your Debian

Overview of the installation steps

1. apt-get install git
2. mkdir -p /var/www/html
3. cd /var/www/html
4. git clone https://framagit.org/hubzilla/core.git .
5. nano .homeeinstall/hubzilla-config.txt
6. cd .homeeinstall/
7. ./hubzilla-setup.sh
8. Reload apache2 service
9. Open your domain with a browser and go through the initial configuration of Hubzilla.

Installation using Docker

It is possible to install Hubzilla comfortably and conveniently as a Docker container. Saiwal (sk@hub.utsukta.org) offers a preconfigured environment for a Hubzilla container for this purpose.

The key features are:

• Use Docker Compose to set up a fully functional Hubzilla instance with just a few commands.
• Prebuilt images available via dockerhub for amd64, arm/v7, arm64.
• Continuous Updates: The Docker image is built to allow for easy updates whenever new changes are made to the Hubzilla core or its dependencies.
• SMTP Integration: Built-in support for sending emails using ssmtp, making it easy to configure email notifications for your Hubzilla instance.

The repository for the container is located here: skprg/hubzilla-docker

Building the image from scratch

• Clone the Repository:

git clone https://github.com/skprg/hubzilla-docker.git
cd hubzilla-docker

• Configure Your Environment: Update the docker-compose.yml file with your SMTP and other settings.
• Build and Run the Container:

docker-compose up --build -d

Using prebuilt image

• Replace the following lines in docker-compose.yml

    build:
      context: .
      dockerfile: Dockerfile 

with

    image: ghcr.io/skprg/hubzilla-docker:latest

• Run the container:

docker compose up -d

Access Your Hubzilla Instance: Navigate to http://localhost (or the appropriate URL) to view your Hubzilla instance.

Further notes / update / tips

Further notes, upgrade instructions and tips can be found in the repository linked above.

Problems Following An Update

A good 90% of all bugs encountered immediately after updating the code to the latest version are simple cache errors of one sort or another. If you update and find something very obvious is broken - like your matrix page doesn't load, notifications are missing, or comment boxes are missing - the chances are it's not a bug at all. Breaking basic functionality is the kind of thing developers tend to notice.

If this happens to you, there are a few simple steps to take before resorting to the support forums:

Browser Cache

Symptoms: Menus do not expand, ACL selector does not open, progress indicator does not display (or loops forever), Matrix and channel pages do not load.

Force reload the page. Shift reload, or ctrl+f5. Occasionally, but very, very rarely, you will also need to clear the session data - which is achieved by restarting the browser.

FastCGI

Symptoms: Incorrect variables. The basic UI mostly works, but displays incorrect content or is missing content entirely.

If you're using php-fpm, this problem is usually resolved with

service php-fpm restart

Smarty Cache

Symptoms:

1) White Screen Of Death. This is most prevalent on the settings and admin pages.

2) Missing icons, tabs, menus or features.

We use the Smarty3 template engine to generate pages. These templates are compiled before they are displayed. Occasionally, a new or modified template will fail to overwrite the old compiled version. To clear the Smarty cache, delete all the files in store/[data]/smarty3/compiled but do not delete the directory itself. Templates will then be recompiled on their next access.

Theme Issues

There are many themes for Hubzilla. Only Redbasic is officialy supported by the core developers. This applies even if a core developer happens to support an additional theme. This means new features are only guaranteed to work in Redbasic.

Redbasic uses a few javascript libraries that are done differently, or entirely absent in other themes. This means new features may only work properly in Redbasic. Before reporting an issue, therefore, you should switch to Redbasic to see if it exists there. If the issue goes away, this is not a bug - it's a theme that isn't up to date.

Should you report an issue with the theme developers then? No. Theme developers use their themes. Chances are, they know. Give them two or three days to catch up and then report the issue if it's still not fixed. There are two workarounds for this situation. Firstly, you can temporarily use Redbasic. Secondly, most themes are open source too - open a pull request and make yourself a friend.

Federation Addons

Various web communities have started to join together using common protocols. The protocols involved are somewhat limited in their possibilities. The Diaspora protocol is somewhat restrictive in terms of the types of communication allowed. All comments must be signed by the original author in a very unique way. The ActivityPub protocol is also supported. No other existing protocol supports nomadic locations (there is now support for nomadic identities in ActivityPub) as used in this project. This poses some challenges for support, as some features work with some networks and not with others. Nonetheless, federation protocols make it possible to connect to a much larger community of people worldwide. They are provided as add-ons.

• Diaspora Protocol - The Diaspora Protocol is used by Diaspora and Friendica. You should first enable ‘Diaspora Statistics’ to utilise all available features.
• PubCrawl - The ActivityPub protocol supported by Mastodon, MIsskey, Friendica and all Fediverse services.

Each member of your site must decide for themselves whether or not to allow these protocols, as they may conflict with some desirable core features and capabilities of this software (such as channel migration and cloning). You do this on your ‘Settings -> Feature/Addon Settings’ page.

Service classes

You can use service classes to limit system resources by restricting the options of individual accounts, e.g. the storage of files and the limitation of posts at the top level. Define customised service classes according to your requirements in the .htconfig.php file. For example, create a standard and a premium class with the following lines:

// Service classes

App::$config[‘system’][‘default_service_class’]=‘standard’; // this is the default service class that will be associated with each new account

// Configuration for the default service class
App::$config[‘service_class’][‘default’] =
array(‘photo_upload_limit’=>2097152, // total storage limit for photos per channel (here 2MB)
‘total_identities’ =>1, // Number of channels an account can create
‘total_items’ =>0, // Number of top-level items a channel can create. Applies only to top-level posts of the channel user, other posts and comments are not affected
‘total_pages’ =>100, // Number of pages a channel can create
‘total_channels’ =>100, // Number of channels the user can add, other users can still add this channel even if the limit is reached
‘attach_upload_limit’ =>2097152, // total storage space for attachments per channel (here 2MB)
‘chatters_inroom’ =>20);

// Configuration for premium service class
App::$config[‘service_class’][‘premium’] =
array(‘photo_upload_limit’=>20000000000, // total storage limit for photos per channel (here 20GB)
‘total_identities’ =>20, // Number of channels an account can create
‘total_items’ =>20000, // Number of top-level items a channel can create. Only applies to top-level posts of the channel user, other posts and comments are not affected
‘total_pages’ =>400, // Number of pages that a channel can create
‘total_channels’ =>2000, // Number of channels the user can add, other users can still add this channel even if the limit is reached
‘attach_upload_limit’ =>20000000000, // Total storage space for attachments per channel (here 20GB)
‘chatters_inroom’ =>100);

To apply a service class to an existing account, use the command line utility from the web root:

util/service_class list service classes

util/config system default_service_class firstclass sets the default service class to ‘firstclass’

util/service_class firstclass lists the services that are part of the ‘firstclass’ service class

util/service_class firstclass photo_upload_limit 10000000 sets the total photo disc usage of the firstclass to 10 million bytes

util/service_class --account=5 firstclass sets the account ID 5 to the service class ‘firstclass’ (with confirmation)

util/service_class --channel=blogchan firstclass sets the account to which the channel ‘blogchan’ belongs to the service class ‘firstclass’ (with confirmation)

Options for limiting the service class

• photo_upload_limit - maximum total number of bytes for photos
• total_items - maximum total number of top-level posts
• total_pages - maximum number of pages for Comanche
• total_identities - maximum number of channels owned by the account
• total_channels - maximum number of connections
• total_feeds - maximum number of RSS feed connections
• attach_upload_limit - maximum file upload space (bytes)
• minimum_feedcheck_minutes - lowest permissible setting for querying RSS feeds
• chatrooms - maximum number of chatrooms
• chatters_inroom - maximum number of chatters per room
• access_tokens - maximum number of guest access tokens per channel

Theme management

Example for repo management

1. Navigate to your hub web rootroot@hub:/root# cd /var/www

2. Add the theme repo and give it a nameroot@hub:/var/www# util/add_theme_repo https://github.com/DeadSuperHero/redmatrix-themes.git DeadSuperHero

Creating Page Templates

A page template for use with Comanche requires two files - a PHP template and a CSS file. Page templates will need to be installed by the system administrator of your site.

First choose a name. Here we'll create a template and call it "demo".

You will need to create the files "view/php/demo.php" and "view/css/demo.css" to hold the PHP template and CSS respectively.

To get a better idea of this process, let's look at an existing template - the "default" template. This is used by default throughout the application.

view/php/default.php

<!DOCTYPE html >
<html>
<head>
    <title><?php if(x($page,'title')) echo $page['title'] ?></title>
    <script>var baseurl="<?php echo z_root() ?>";</script>
    <?php if(x($page,'htmlhead')) echo $page['htmlhead'] ?>
</head>
<body>
    <?php if(x($page,'nav')) echo $page['nav']; ?>
    <aside id="region_1"><?php if(x($page,'aside')) echo $page['aside']; ?></aside>
    <section id="region_2"><?php if(x($page,'content')) echo $page['content']; ?>
        <div id="page-footer"></div>
        <div id="pause"></div>
    </section>
<aside id="region_3"><?php if(x($page,'right_aside')) echo $page['right_aside']; ?></aside>      
<footer><?php if(x($page,'footer')) echo $page['footer']; ?></footer>
</body>
</html>    

Here's is the corresponding CSS file

view/php/default.css

aside#region_1 {
    display: block;
    width: 210px;
    position: absolute;
    top: 65px;
    left: 0;
    margin-left: 10px;
}

aside input[type='text'] {
    width: 174px;
}


section {
    position: absolute;
    top: 65px;
    left: 250px;
    display: block;
    right: 15px;
    padding-bottom: 350px;
}

Some things you may notice when looking at these definitions:

• We have not specified any CSS for the "nav", "right_aside", or "footer" regions. In this template "nav" and "footer" will be the full page width and we will let the size and placement of these elements be controlled by the theme. "right_aside" is not currently used.

• There are elements on the page such as "page-footer" and "pause" for which there is no apparent content. This content will come from Javascript elements.

• Our default template uses absolute positioning. Modern web design often uses "float" div containers so that scrollbars aren't typically needed when viewing on small-screen devices.

To design a new template, it is best to start with an existing template, and modify it as desired. That is what we will do here.

The way that Comanche provides content inside a specific region is by using a region tag.

[region=aside][widget=profile][/widget][/region]

This example will place a "profile" widget in the "aside" region. But what it actually does is place the HTML for the widget into a code variable $page['aside']. Our default page template defines a region on the page (the CSS positions this as an absolute sidebar) and then inserts the contents of $page['aside'] (if it exists).

So if you wanted to create a template with a region named "foo", you would provide a place for it on the page, then include the contents of $page['foo'] wherever you wanted to use it, and then using Comanche, you could specify

[region=foo][widget=profile][/widget][/region]

and this would place a profile widget into the "foo" region you created.

Use the CSS file to position the region on the page where desired and optionally control its size.

Hubzilla development - a guide to the schema system

A schema, in a nutshell, is a collection of settings for a bunch of variables to define certain elements of a theme. A schema is loaded as though it were part of config.php and has access to all the same information. Importantly, this means it is identity aware, and can be used to do some interesting things. One could, for example, restrict options by service class, or present different options to different members.

By default, we filter only by whether or not expert mode is enabled. If expert mode is enabled, all options are presented to the member. If it is not, only scheme, background image, font face, and iconset are available as choices.

A schema is loaded after the member's personal settings. Therefore, to allow a member to overwrite a particular aspect of a schema you would use the following syntax:

    if (! $foo)
        $foo = 'bar';

However, there are circumstances - particularly with positional elements - where it may be desirable (or necessary) to override a member's settings. In this case, the syntax is even simpler:

        $foo = 'bar';

Members will not thank you for this, however, so only use it when it is required.

If no personal options are set, and no schema is selected, we will first try to load a schema with the file name "default.php". This file should never be included with a theme. If it is, merge conflicts will occur as people update their code. Rather, this should be defined by administrators on a site by site basis. default.php and default.css MUST be symlinks to existing scheme files.

You schema does not need to - and should not - contain all of these values. Only the values that differ from the defaults should be listed. This gives you some very powerful options with very few lines of code.

Note the options available differ with each theme. The options available with the Redbasic theme are as follows:

• nav_colour The colour of the navigation bar. Options are red, black and silver. Alternatively, one can set $nav_bg_1, $nav_bg_2, $nav_bg_3 and $nav_bg_4 to provide gradient and hover effects.
• banner_colour The font colour of the banner element. Accepts an RGB or Hex value.
• bgcolour Set the body background colour. Accepts an RGB or Hex value.
• background_image Sets a background image. Accepts a URL or path.
• item_colour Set the background colour of items. Accepts an RGB or Hex value.
• item_opacity Set the opacity of items. Accepts a value from 0.01 to 1
• toolicon_colour Set the colour of tool icons. Accepts an RGB or Hex value.
• toolicon_activecolour Set the colour of active or hovered icon tools.
• font_size Set the size of fonts in items and posts. Accepts px or em.
• body_font_size Sets the size of fonts at the body level. Accepts px or em.
• font_colour Sets the font colour. Accepts an RGB or Hex value.
• radius Set the radius of corners. Accepts a numeral, and is always in px.
• shadow Set the size of shadows shown with inline images. Accepts a numerical value. Note shadows are not applied to smileys.
• converse_width Set the maximum width of the content region in px.
• nav_min_opacity
• top_photo
• reply_photo

If a your_schema_name.css file is found, the content of this file will be attached to the end of style.css. This gives the schem developer the possiblity to override any style component.

Core Widgets

Some/many of these widgets have restrictions which may restrict the type of page where they may appear or may require login

• clock - displays the current time

  • args: military (1 or 0) - use 24 hour time as opposed to AM/PM

• profile - displays a profile sidebar on pages which load profiles (pages with nickname in the URL)

• tagcloud - display a tagcloud of webpage items

  • args: count - number of items to return (default 24)

• collections - privacy group selector for the current logged in channel

  • args: mode - one of "conversation", "group", "abook" depending on module

• suggestions - friend suggestions for the current logged on channel

• follow - presents a text box for following another channel

• notes - private notes area for the current logged in channel if private_notes feature is enabled

• savedsearch - network/matrix search with save - must be logged in and savedsearch feature enabled

• filer - select filed items from network/matrix stream - must be logged in

• archive - date range selector for network and channel pages

  • args: 'wall' - 1 or 0, limit to wall posts or network/matrix posts (default)

• fullprofile - same as profile currently

• categories - categories filter (channel page)

• tagcloud_wall - tagcloud for channel page only

  • args: 'limit' - number of tags to return (default 50)

• catcloud_wall - tagcloud for channel page categories

  • args: 'limit' - number of categories to return (default 50)

• affinity - affinity slider for network page - must be logged in

• settings_menu - sidebar menu for settings page, must be logged in

• mailmenu - sidebar menu for private message page - must be logged in

• design_tools - design tools menu for webpage building pages, must be logged in

• findpeople - tools to find other channels

• photo_albums - list photo albums of the current page owner with a selector menu

• vcard - mini profile sidebar for the person of interest (page owner, whatever)

• dirsafemode - directory selection tool - only on directory pages

• dirsort - directory selection tool - only on directory pages

• dirtags - directory tool - only on directory pages

• menu_preview - preview a menu - only on menu edit pages

• chatroom_list - list of chatrooms for the page owner

• bookmarkedchats - list of bookmarked chatrooms collected on this site for the current observer

• suggestedchats - "interesting" chatrooms chosen for the current observer

• item - displays a single webpage item by mid or page title

  • args:
  • channel_id - channel that owns the content, defualt is the profile_uid
  • mid - message_id of webpage to display (must be webpage, not a conversation item)
  • title - URL page title of webpage (must provide one of either title or mid)

• photo - display a single photo

  • args:
  • src - URL of photo, must be http or https
  • zrl - use zid authenticated link
  • style - CSS style string

• cover_photo - display the cover photo for the selected channel

  • args:
  • channel_id - channel to use, default is the profile_uid
  • style - CSS style string (default is dynamically resized to width of region)

• photo_rand - display a random photo from one of your photo albums. Photo permissions are honoured

  • args:
  • album - album name (very strongly recommended if you have lots of photos)
  • scale - typically 0 (original size), 1 (1024px), 2, (640px), or 3 (320px)
  • style - CSS style string
  • channel_id - if not your own

• random_block - display a random block element from your webpage design tools collection. Permissions are honoured.

  • args:
  • contains - only return blocks which include the contains string in the block name
  • channel_id - if not your own

• tasklist - provide a task or to-do list for the currently logged-in channel.

  • args:
  • all - display completed tasks if all is non-zero.

• forums - provide a list of connected public forums with unseen counts for the current logged-in channel.

• activity - provide a list of authors of unread network content for the current logged-in channel.

• album - provides a widget containing a complete photo album from albums belonging to the page owner; this may be too large to present in a sidebar region as is best implemented as a content region widget.

  • args:
  • album - album name
  • title - optional title, album name is used if not present

Creating New Widgets

Class Widgets

To create a class-based widget named 'slugfish' create a file with the following contents:

<?php

namespace Zotlabs\Widget;


class Slugfish {

    function widget($args) {

    ... widget code goes here.
    ... The function returns a string which is the HTML content of the widget.
    ... $args is a named array which is passed any [var] variables from the layout editor
    ... For instance [widget=slugfish][var=count]3[/var][/widget] will populate $args with
    ... [ 'count' => 3 ]

    }

The resultant file may be placed in widget/Slugfish/Slugfish.php , or Zotlabs/SiteWidgets/Slugfish.php . It also may be linked from a git repository using util/add_widget_repo.

Traditional function based widget:

If you want a widget named 'slugfish', create widget/widget_slugfish.php containing

<?php

function widget_slugfish($args) {

.. widget code goes here. See above information for class-based widgets for details.

}

Primary Directory

By default, hubzilla will use available Directories on the web, which show you channels available around the world.

There are certain scenarios where you might want your own directory-server that you can connect multiple hubs to. This will limit the channels that appear in all of your hubs to only channels on hubs connected to your directory-server.

Instuctions on how to set up one hub as the Primary Directory for a series of private hubs.

---

• On the hub that will be the Directory Server, open the .htconfig.php file and set:

  App::$config['system']['directory_mode'] = DIRECTORY_MODE_PRIMARY;

  By default it should already be set as DIRECTORY_MODE_NORMAL, so just edit that line to say DIRECTORY_MODE_PRIMARY

• Next, for each hub (including the Directory Server), from a terminal, cd into the folder where it is installed and run this :

  util/config system directory_realm YOURREALMNAME

  (YOURREALMNAME can be whatever you want your realm-name to be)

  then:

  util/config system realm_token THEPASSWORD

  (THEPASSWORD is whatever password you want for your realm)

  NOTE: Use the same realm-name and password for each hub

• Lastly, for each "client" hub, (from a terminal) run:

  util/config system directory_server https://theaddressofyourdirectoryserver.com

---

Now when you view the directory of each hub, it should only show the channels that exist on the hubs in your realm. I have tested with two hubs so far, and it seems to be working fine. Channels created in each hub are reflected in the Primary Directory, and subsequently in the directory of all client hubs

Issues

---

When I created the first hub,it was up and running for an hour or so before I changed it to PRIMARY_MODE, and after changing it, there were a few channels from across the matrix still present in the directory. I deleted them from the xchan table and that seems to have fixed the issue.

Administration

Website administration

The website is usually managed via the administration page, which is located under /admin on your website. To access this page, you must have administration rights for the server. Administration rights are granted to the first account that registers on your website, provided that the email address of this account exactly matches the email address that you specified as the administrator's email address during setup. There are several ways in which this can fail and the system can be left without an administrator account, for example if the first account created has a different email address to the administrator email address specified during setup.

For security reasons, there is no web page or interface on the system that gives you administrator rights. If you need to correct a situation where a system does not have an administrator account, this must be done by editing the account table in the database. There is no other option. To do this, you must find the entry in the account table that belongs to the desired administrator and set ‘account_roles’ for this entry to 4096. You can then call up the administrator page via the profile menu of your system or directly via /admin. A hub can have several administrators and the number of administrators is not limited. Repeat the above procedure for each account that you want to give administration rights.

Troubleshooting

Log files

The system log file is an extremely useful resource for tracking down errors. This can be activated on the admin/log configuration page. A log level setting of LOGGER_DEBUG is preferred for stable production sites. Most problems with communication or storage are listed here. A setting of LOGGER_DATA provides much more detail, but can fill up your hard drive. In any case, we recommend using logrotate on your operating system to check the logs and delete older entries.

At the end of the .htconfig.php file there are several (commented out) lines that enable PHP error logging. This reports problems with the code syntax and execution of the code and is the first place you should look for problems that result in a ‘white screen’ or blank page. This is usually the result of code/syntax problems. Database errors are reported in the system log file, but we have found it useful to create a file called dbfail.out in your root directory where only database related problems are collected. If the file exists and is writable, database errors will be logged in this file as well as in the system log file.

In the case of ‘500’ errors, the problems are often logged in the logs of your web server, often in /var/log/apache2/error.log or similar. Consult the documentation of your operating system. There are three different logging options.

The first is the database error log.

This is only used if you create a file with the name dbfail.out in the root directory of your website and make it writable for the web server. If database queries have failed, they are reported here. They generally indicate typing errors in our queries, but also occur when the database server interrupts the connection or tables become corrupted. In rare cases, race conditions are reported here when two processes try to create an xchan or cache entry with the same ID. All other errors (especially persistent errors) should be investigated.

The second area is the PHP error log.

This is generated by the language processor and only reports problems in the language environment. Again, these can be syntax or programming errors, but they are usually fatal and result in a ‘white screen of death’, i.e. PHP is terminated. You should look at this file if something goes wrong that does not result in a white screen, but it is not uncommon for this file to be blank for days.

At the end of the supplied .htconfig.php file are some lines that, if not commented out, will enable a PHP error log (extremely useful for finding the cause of white screen errors). This is not the case by default as there may be issues with log file ownership and write permissions and the log file does not rotate by default.

The third file is the ‘application log’.

This is used by Hubzilla to report what is going on in the programme and usually reports any difficulties or unexpected data we have received. Occasionally, ‘heartbeat’ status messages are also output to indicate that we have reached a certain point in a script. This is the most important log file for us, as we create it ourselves for the sole purpose of reporting the status of background tasks and anything that seems strange or out of place. It doesn't necessarily have to be something bad, but maybe just unexpected. If you are running a task and a problem occurs, let us know what is in this file when the problem occurs. (Please don't send me 100M dumps, that would just annoy me). Just a few relevant lines so I can eliminate a few hundred thousand lines of code and focus on where the problem is occurring.

These are your website logs, not mine. We report serious issues at every log level. For most websites, I recommend the DEBUG log level - it provides a little extra information and doesn't generate huge log files. If a problem occurs that defies all attempts to track it, you should use the DATA log level for a short period of time to capture all the details of the structures we were dealing with at the time. This log level consumes a lot of memory and is therefore only recommended for short periods of time or for developer test sites. I recommend configuring logrotate for both the php log and the application log. I usually take a look at dbfail.out every week or two, fix any reported issues and then start over with a new file. The same goes for the PHP log file. I check it from time to time to see if there is anything that needs to be fixed.

If something goes wrong and it's not a serious error, I look at the application log file. Often I will run

tail -f logfile.out

while repeating a process where there are problems. Often I'll add additional logging statements to the code if there's no indication of what's going wrong. Even something as simple as ‘got here’ or expressing the value of a variable that might be suspicious. You can do this too - in fact, I would encourage you to do so. Once you've found what you need to look for, you can run

git checkout file.php

to immediately delete any extra logging data you've added. Use the information from this log and any details you can provide from your investigation of the problem to file your bug report - unless your analysis points to the source of the problem. In this case, simply fix it.

Rotating log files

1. Activate the logrot addon in the official hubzilla-addons repo
2. Create a directory in your web root called log with write permissions for the web server
3. Go to the logrot admin settings and enter this folder name and the maximum size and number of log files kept.

Report problems

When reporting problems, please try to provide as much detail as necessary to allow the developers to reproduce the problem and include the full text of any error messages.

We encourage you to use these logs, along with the source code you have, to the best of your ability to fix problems and find their cause. The community can often help, but only you have access to your site's log files, and it is considered a security risk to share them.

If a code issue has been uncovered, please report it in the project's bug tracker (https://framagit.org/hubzilla/core/issues). Again, provide as much detail as possible so we don't have to keep asking questions about your configuration or duplicating the issue so we can address the problem directly and figure out what to do. You are also welcome to offer your own solutions and submit patches. In fact, we encourage you to do so, as we are all volunteers and have limited time available. The more people that help out, the easier the work will be for everyone. It's okay if your solution isn't perfect. Every little bit helps, and maybe we can improve it.

Hub snapshot tools

Hubzilla developers often need to switch between branches that may have incompatible database schemas or content. The following two scripts create and restore complete snapshots of a Hubzilla instance, including the Hub web root and the entire database state. Each script requires a configuration file called hub-snapshot.conf, which is located in the same folder and contains the specific directories and database details of your Hub.

Configuration file

The format of the configuration file is very strict. There must be no spaces between the variable name and the value. Replace only the content inside the inverted commas with your configuration. Save this file as hub-snapshot.conf together with the scripts.

# Location of the hub root. Normally this is the location of the Hubzilla repo clone.
HUBROOT=‘/var/www/’
# Name of the MySQL database
DBNAME=‘hubzilla’
# MySQL database user
DBUSER=‘hubzilla’
# MySQL database password
DBPWD=‘akeufajeuwfb’
# The target snapshot folder in which the Git repository is to be initialised
SNAPSHOTROOT=‘/root/snapshots/hubzilla/’

Snapshot

Example of use:

sh hub-snapshot.sh my-hub.conf ‘Commit message for the snapshot’ 

hub-snapshot.sh:

#!/bin/bash

if ! [ -f ‘$1’ ]; then
    echo ‘$1 is not a valid file. Aborting..."
    exit 1
fi
source ‘$1’
#echo ‘$DBNAME’
#echo ‘$DBUSER’
#echo ‘$DBPWD’
#echo ‘$HUBROOT’
#echo ‘$SNAPSHOTROOT’
MESSAGE=‘snapshot: $2’

if [ ‘$DBPWD’ == ‘’ -o ‘$SNAPSHOTROOT’ == ‘’ -o ‘$DBNAME’ == ‘’ -o ‘$DBUSER’ == ‘’ -o ‘$HUBROOT’ == ‘’ ]; then
    echo ‘Required variable is not set. Aborting..."
    exit 1
fi

if [ ! -d ‘$SNAPSHOTROOT’/db/ ]; then
    mkdir -p ‘$SNAPSHOTROOT’/db/
fi
if [ ! -d ‘$SNAPSHOTROOT’/www/ ]; then
    mkdir -p ‘$SNAPSHOTROOT’/www/
fi

if [ ! -d ‘$SNAPSHOTROOT’/www/ ] || [ ! -d ‘$SNAPSHOTROOT’/db/ ]; then
    echo ‘Error creating snapshot directories. Aborting..."
    exit 1
fi

echo ‘Export database...’
mysqldump -u ‘$DBUSER’ -p‘$DBPWD’ ‘$DBNAME’ > ‘$SNAPSHOTROOT’/db/‘$DBNAME’.sql
echo ‘Copy hub root files...’
rsync -va --delete --exclude=.git* ‘$HUBROOT’/ ‘$SNAPSHOTROOT’/www/

cd ‘$SNAPSHOTROOT’

if [ ! -d ‘.git’ ]; then
    git init
fi
if [ ! -d ‘.git’ ]; then
    echo ‘Cannot initialise git repo. Aborting..."
    exit 1
fi

git add -A
echo ‘Commit hub snapshot...’
git commit -a -m ‘$MESSAGE’

exit 0

Restore

/bin/bash
# Restore hub to a previous state. Input hub config and commit hash

if ! [ -f ‘$1’ ]; then
        echo ‘$1 is not a valid file. Aborting..."
        exit 1
fi
source ‘$1’
COMMIT=$2

if [ ‘$DBPWD’ == ‘’ -o ‘$SNAPSHOTROOT’ == ‘’ -o ‘$DBNAME’ == ‘’ -o ‘$DBUSER’ == ‘’ -o ‘$HUBROOT’ == ‘’ ]; then
        echo ‘Required variable is not set. Aborting..."
        exit 1
fi
RESTOREDIR=‘$(mktemp -d)/’

if [ ! -d ‘$RESTOREDIR’ ]; then
    echo ‘Cannot create restore directory. Aborting..."
    exit 1
fi
echo ‘Cloning the snapshot repo...’
git clone ‘$SNAPSHOTROOT’ ‘$RESTOREDIR’
cd ‘$RESTOREDIR’
echo ‘Checkout requested snapshot...’
git checkout ‘$COMMIT’
echo ‘Restore hub root files...’
rsync -a --delete --exclude=.git* ‘$RESTOREDIR’/www/ ‘$HUBROOT’/
echo ‘Restore hub database...’
mysql -u ‘$DBUSER’ -p‘$DBPWD’ ‘$DBNAME’ < ‘$RESTOREDIR’/db/‘$DBNAME’.sql

chown -R www-data:www-data ‘$HUBROOT’/{store,extend,addon,.htlog,.htconfig.php}

echo ‘Restored hub to snapshot $COMMIT’
echo ‘Removing temporary files...’

rm -rf ‘$RESTOREDIR’

exit 0

Database

Database updates

On the https://hub.hubzilla.hu/admin/dbsync page, the administrator can check whether an update has failed and retry it if necessary. If an update has failed, but for some reason is not registered as failed, the administrator can try to run the update again. For example, for DB update #1999, by visiting the web page: https://hubzilla.com.bradmin/dbsync/1999

Database tables

Table	Description
abconfig	any storage space for connections from local channels
abook	Connections of the local channels
account	Service provider account
addon	registered plugins
app	personal app data
attach	File attachments
auth_codes	OAuth usage
cache	OEmbed cache
cal	CalDAV container for events
channel	local channels
chat	Chat room content
chatpresence	Channel presence information for the chat
chatroom	Data for the actual chat room
clients	OAuth usage
config	Main configuration storage
conv	Meta conversation structure for private messages in Diaspora
event	Events
pgrp_member	Data protection groups (collections), group information
pgrp	Data protection groups (collections), member information
hook	Plugin hook register
hubloc	xchan location storage, links a hub location with a xchan
iconfig	Expandable, arbitrary memory for elements
issue	Upcoming error/problem database
item	all articles and websites
item_id	(deprecated by iconfig) other identifiers in other services for contributions
likes	‘Like‘ things
mail	private messages
menu	Website menu data
menu_item	Entries for menus on websites
notify	Notifications
obj	Object data for things (x has y)
outq	Outgoing queue
pconfig	Personal (per channel) configuration memory
photo	Photo store
poll	Data for polls
poll_elm	Data for poll elements
profdef	Definitions for user-defined profile fields
profext	User-defined profile field data
profile	Channel profiles
profile_check	DFRN remote authorisation, may be outdated
register	Registrations that require an administrative authorisation
session	Storage of web sessions
shares	Information on common elements
sign	Diaspora signatures. Is gradually being phased out.
site	Location table for finding directory peers
source	Data from channel sources
sys_perms	Extensible authorisations for OAuth
term	Article taxonomy table (categories, tags, etc.)
tokens	OAuth usage
updates	Directory synchronisation updates
verify	General verification structure
vote	Voting data for polls
xchan	List of known channels in the universe
xchat	Chat rooms with bookmarks
xconfig	like pconfig, but for channels without a local account
xign	Channels ignored by friend suggestions
xlink	‘Friends of friends’ links derived from poco, also storage of ratings
xperm	OAuth/OpenID-Connect extendable authorisations Authorisation memory
xprof	if this node is a directory server, it contains basic public profile information about everyone on the network
xtag	if this hub is a directory server, it contains tags or interests from everyone on the network

Advanced configurations for administrators

This document assumes that you are an administrator. Hubzilla contains many configuration options that are hidden in the main administration area. Generally, these are options that are considered too niche, advanced, or confusing. These settings can be changed via the shell from the top level web directory using the following syntax

util/config cat key value

for a website configuration, or

util/pconfig channel_id cat key value

for a member configuration.

For a site configuration, another option is to insert a line in .htconfig.php with the syntax:

App::$config[‘cat’][‘key’] = ‘value’;`

Member configuration (pconfig)

• system.always_my_theme Always use your own theme when watching channels in the same hub. This leads to some pretty imaginative problems when viewing channels with themed comanches.
• system.blocked An array of Xchans that are blocked by this channel. Technically this is a hidden configuration and belongs here, but addons (notably Superblock) have made this available in the UI.
• system.default_cipher Sets the default cipher used for E2EE elements.
• system.display_friend_count Defines the number of connections to be displayed in the connection profile widget.
• system.do_not_track Like the browser header. This will break many identity-based functions. You should really only set permissions that make sense.
• system.forcepublicuploads Forces uploaded photos to be public when uploaded as wall elements. It makes much more sense to set the permissions correctly in the first place. Do this instead.
• system.network_page_default Sets the default parameters for displaying the network page. This should contain the same query string as the manual filtering.
• system.paranoia Defines the security level of the IP check. If the IP address of a logged on session changes, this level is applied to determine if the account should be logged off as a security breach. The options are: 0 - no IP check 1 - check 3 octets 2 - check 2 octets 3 - check for all differences
• system.prevent_tag_hijacking Prevents external networks from hijacking hashtags in your posts and redirecting them to their own resources.
• system.startpage Another one of those technically hidden configurations provided by addons. Defines the default page that is displayed when you log in. This is made available to the user interface by the startpage add-on.
• system.taganyone Requires that the configuration of the same name is activated. Allows @mention tagging of everyone, regardless of whether you are connected or not. This does not scale.
• system.anonymous_comments By default or if set to 1, custom permissions can be set to allow anonymous (moderated) comments like WordPress, moderated by the channel owner. If set to 0, no member of your site can select or enable this.
• system.user_scalable Determines whether the app is scalable to touchscreens. Set to on by default, set to zero to deactivate - real zero, not just false.

Configuration of the website

• randprofile.check When requesting a random profile, first check whether it actually exists
• randprofile.retry Number of attempts to obtain a random profile
• system.admin_email Specifies the e-mail address of the administrator for this site. This is defined during installation.
• system.authlog Log file used for logging authentication errors. Used to connect to server-side software such as fail2ban. Auth errors are also logged in the main logs.
• system.auto_channel_create Adds the necessary form elements to create the first channel on the account registration page and creates it (possibly after email validation or administrator approval). This excludes the ability to import a channel from another site as the first channel created on this site for a new account. Use with system.default_permissions_role to streamline registration.
• system.auto_follow The first channel of an account automatically follows the channels listed here - comma separated list of webbies (member@hub addresses).
• system.blacklisted_sites An array of specific hubs that are to be completely blocked by this hub.
• system.block_public_search Similar to block_public, with the difference that only public access to the search functions is blocked. Useful for pages that want to be public but are overrun by search engines.
• system.cron_hour Specify an hour in which cron_daily should be executed. By default, without configuration, this is executed at midnight UTC.
• system.default_permissions_role If this value is set to a valid name for a permission role, this role will be used for the first channel created by a new account and will not prompt for the ‘channel type’ on the channel creation form. Examples of valid names are: ‘social’, ‘social_restricted’, ‘social_private’, ‘forum’, ‘forum_restricted’ and ‘forum_private’. Read more about authorisation roles here.
• system.default_profile_photo Defines the profile photo with which new channels start. This should contain the name of a directory located under images/default_profile_photos/ or it should not be set. If it is not set, ‘rainbow_man’ is assumed.
• system.directorytags Specifies the number of keyword tags to display on the directory page. The default is 50 if it is not set to a positive integer.
• system.disable_directory_keywords If ‘1’, no directory keywords are displayed. If the hub is a directory server, prevent keywords from being returned to all directory clients. Please do not set this for directory servers in the RED_GLOBAL area.
• system.disable_discover_tab Allows you to completely disable the ability to discover public content from external sites.
• system.disable_dreport If ‘1’, no delivery reports are saved or linked.
• system.dlogfile Log file used for logging development errors. Exactly the same as logger otherwise. This is not magic and requires your own logging instructions. Developer tool.
• system.email_notify_icon_url URL of the image (32x32) to be displayed in email notifications (HTML bodies).
• system.expire_delivery_reports Expiry date in days for delivery reports - default value 10
• system.expire_limit Do not expire more than this number of reports per channel per expiry run to avoid exhausting the memory. Default value 5000.
• system.photo_storage_type If ‘1’, the file system is used instead of the SQL database to store the thumbnails. Default setting is ‘0’. Introduced in 4.2
• system.hidden_version_siteinfo If ‘true’, the software version is not displayed on the siteinfo pages (system.hide_version hides the version on these pages as well, this setting only hides the version on the siteinfo pages).
• system.hide_help Do not show the link to the help documentation in the navigation bar
• system.hide_in_statistics Instructs the red statistics servers to completely hide this hub in hub lists.
• system.hide_version If true, the software version is not displayed on websites and tools. (*) Must be set in .htconfig.php.
• system.ignore_imagick Ignores imagick and uses GD, even if imagick is installed on the server. Prevents some problems with PNG files in older versions of imagick.
• system.max_daily_registrations Defines the maximum number of new registrations allowed in a day. Useful to prevent oversubscription when publicising the project.
• system.max_import_size If configured, the maximum length of an imported text message. This is usually left at 200 KBytes or more to accommodate private Friendica photos that are embedded.
• system.max_tagged_forums Spam protection. Limits the number of tagged forums recognised in each post. Default is 2, only the first ‘n’ tags are delivered as forums, the others cause no delivery.
• system.minimum_feedcheck_minutes The minimum interval between checking RSS feeds. If this interval is less than the cron interval, the feeds are checked with every cron job. The default value is 60 if it has not been set. The site setting can also be overridden for each individual channel by a service class setting aptly named ‘minimum_feedcheck_minutes’.
• system.no_age_restriction Do not restrict registration to persons over the age of 13. In many countries it is required by law that age must be specified and that all personal data of minors must be blocked.
• system.object_cache_days Defines how long cached embedded content can be used without being retrieved again. The default setting is 30 days.
• system.openssl_conf_file Specify a file that contains the OpenSSL configuration. Required in some Windows installations to find the openssl configuration file on the system. Read the code first. If you cannot read the code, do not play with it.
• system.openssl_encrypt Use encryption engine from openssl, default is false (uses mcrypt for AES encryption)
• system.optimize_items Executes optimise_table during some tasks to keep your database clean and defragmented. This comes at the expense of performance while the operations are running, but also makes things run a bit faster when they are not. There are also CLI utilities to perform this operation that you may prefer, especially if you have a large site.
• system.override_poll_lockfile Ignores the lock file in the poller process to allow more than one process to run at a time.
• system.paranoia Like pconfig, but on a site-wide basis. Can be overridden by member settings.
• system.pin_types Array of allowed element types for pinning. The default values depend on the module, but can be changed here.
• system.photo_cache_time How long the photos should be cached, in seconds. The default value is 86400 (1 day). A longer time increases performance, but also means that it takes longer for changed authorisations to apply.
• system.platform_name What should be displayed as the platform name on websites and in statistics. (*) Must be set in .htconfig.php.
• system.rating_enabled Distributed reputation reports and data collection. This function is currently being revised.
• system.poke_basic Reduce the number of poke verbs to exactly 1 (‘poke’). Deactivate other verbs.
• system.proc_run_use_exec If 1, the system call exec in proc_run is used to execute background tasks. By default we use proc_open and proc_close. On some (currently rare) systems this does not work well.
• system.projecthome Displays the project page on your homepage for logged out viewers.
• system.projecthome Defines the project home page as the start page of the hub. (Obsolete)
• system.pubstream_order Defines the pubstream order. Possible values ‘commented’ (default), ‘created’ and ‘edited’.
• system.register_link Path to which the ‘register’ link in the registration form should point. For closed sites, this will point to ‘pubsites’. For open sites, it will normally redirect to ‘register’, but you can change this to a custom site page that offers subscriptions or similar.
• system.reserved_channels Do not allow members to register channels with this comma-separated list of names (no spaces).
• system.sellpage A URL to display in the list of public sites to sell your hub - show classes of service, etc.
• system.startpage Defines the default page that is displayed for all channels on this website after a login. Can be overridden by user settings.
• system.sys_expire_days How many days should discovered public content from other websites be kept?
• system.taganyone Allows @mention tagging of everyone, whether you are connected or not.
• system.tempdir Location where temporary files are stored (currently unused), default is defined in PHP configuration.
• system.tos_url Sets an alternative link for the ToS storage location.
• system.transport_security_header if non-zero and SSL is used, a strict-transport-security header is inserted on the web pages
• system.uploaddir Storage location for uploading files (default is system.tempdir, currently only used by the js_upload plugin)
• system.workflow_channel_next The page to which new members should be redirected immediately after creating a channel.
• system.workflow_register_next The page to which members are redirected immediately after creating an account (only if auto_channel_create or UNO is activated).

Directory configuration

Default values for the directory search

• directory.globaldir 0 or 1. default value 0. If you visit the directory on a site, you will only see the members of this site by default. You have to take an extra step to see the people in the rest of the network; and by doing this, there is a clear delineation that these people are not members of this site, but of a larger network.
• directory.pubforums 0 or 1. public forums should be 0 by default.
• directory.safemode 0 or 1.

Configuration of the directory server

• system.directory_mode
• system.directory_primary
• system.directory_realm
• system.directory_server
• system.realm_token

CLI Tools (utils)

If you have access to the shell as an administrator, you can use other small CLI tools, which can be found in the ‘utils’ directory.

addons

You can use the addons script to display which addons are installed and which are available. You can also install and uninstall add-ons and reinstall all add-ons.

• util/addons list Display of all installed add-ons
• util/addons list all Display of all add-ons that are installed (*) and those that are deactivated due to incompatibility (!)
• util/addons install foo Install the addon with the name ‘foo’
• util/addons uninstall foo Uninstall the addon with the name ‘foo’
• util/addons reinstall Reinstall all addons

admins

The admins script allows you to display all admins of the hub, add further admins and remove existing admins.

• util/admins
• util/admins list
• util/admins add <account_id>
• util/admins remove <account_id>

config / pconfig

See: Advanced configurations for administrators

connect

You can use connect to establish a connection between one channel of your hub and another channel.

• util/connect <channel_id> <channel@hub>
• util/connect <channel_address> <channel@hub>

dcp

You can use dcp to copy files to the store area of a channel on your hub.

• util/dcp <source> <destination directory> where destination directory must be store/$nickname/path or $nickname/path.

dmkdir

You can use dmkdir to create a subdirectory in the store area of a channel on your hub.

• util/dmkdir <directory> where directory must ba store/$nickname/path/<directory> or $nickname/path/<directory>.

fresh (The Freaking REd Shell)

This only works under Unix/Linux. If the readline module is installed, it uses this for the input, otherwise it only reads from stdin and writes to stdout. The commands are processed in sequence until the command ‘exit’, ‘quit’ or the end of the file is reached.

Commands:

• version Displays the current Fresh version.

• login <email address>

• Asks for a password and authenticates <email address> as the current user.

  user.

• finger <channel address> Performs a lookup of <channel address> and reports the result.

• channel <channel nbickname> Switches the current channel to the channel with the specified nickname.

• conn [<id1> <id2> ...]

  Without arguments, this lists all connections of the current channel with an ID. If IDs are specified, the details of the individual connections are displayed.

hz

Simple, minimalist command line tool to send the status to hubzilla via the API. Requires curl.

hz [<configuration file>]

hz requires a configuration file. You can either use a .hubzilla file and omit the <configuration file> parameter or create a configuration file under any name in the hubzilla directory, the name of which you then specify when calling hz.

Format of the configuration file:

USER=<your username>
PASS=<your password>
HUB=<domain of the hub>

You can then enter your posting and finalise the entry with Ctrl-D.

storageconf

You can use storageconf to specify the storage location for thumbnails (file system or database) and query the current configuration.

• util/storageconv stats Shows the currently set status
• util/storageconv fs Moves the thumbnails from the database to the file system
• util/storageconv db Moves the thumbnails from the file system to the database

thumbrepair

thumbrepair recreates the local thumbnails.

File Sync and Clone

File cloning across multiple instances of a channel is a very hard problem, due to the nature of PHP memory allocation. This needs to be handled dramatically differently than cloning or syncing of other information. (Processing one large video file or 40-50 photos could exhaust memory). Therefore we can't easily just dump all the data to a dump file and sequentially process it. Loading the dump file itself is likely to exhaust memory.

There are also two primary operations we are considering. The first is the hardest - saving and then importing all your channel information into a new channel clone. The second is synchronising file changes as they occur across two or more "active" clones.

For the first cut at this tool we will concentrate on the second case, while trying to maintain some measure of compatibility with the first case so that we can re-use the same tools.

Meta Data

First we need the metadata for the file in order to precisely re-construct its structure on another site. This requires the following information:

'attach' structure (without file contents - which is the default) for the file itself and its parent directories so that we can re-create its precise place in the file system, since we do not know if the parent directory has been imported previously or ever.

'photo' structure for any photo elements which were created as a result of uploading this file into the system. This typically contains several different 'scales' or thumbnail images, some of which may be cropped for profile photo use or cover photo use. We need to retain the cropping information which is not present in the metadata, but only in the stored data. The actual thumbnail image data may or may not be included in the metadata. A cover photo of large scale (scale #7) could potentially cause memory issues. Not as bad as a 100M video, but if you have several of these they could add up.

'item' entries which are linked to this file. These can be file share activities, the "parent item" linked to photos, and any attached conversation items (photo likes, comments, etc.)

All of these items will require URL replacement and re-signing of the item as they are relocated to another site.

File Data

Then we have the actual file data we need to reconstruct the file. This needs to be stored separately from the meta-data to avoid memory exhaustion when processing. The actual file data can be used to reconstruct the attach structure and the first four photo scales. If this is a photo, we need access to the "#4 scale" (profile photo) and the #7 scale (cover photo) as they were originally cropped. All other thumbnails can be generated from these.

File Sync

We will consider this operation first because it is probably the most straightforward to implement. When a photo is added to or removed or changed from the source system, we will send a clone sync packet to all known clones containing the metadata - but no file data . We can only send one sync packet per file operation that needs to be synced.

The receiving end will create and perform URL translation on all the metadata structures and store them. Then it will need to fetch the actual data. Assuming CURL supports streaming, an authenticated request is sent to the original site and the original file is requested and streamed directly to disk (bypassing all processing). If photo scale #4 or scale #7 is required, these are requested and stored into their respective structures. We're assuming in this case that the cover photo large scale will not exhaust memory. If CURL cannot be made to support streaming, request packets need to be queued and sent to the origination site to obtain "chunks" of the file and re-assembled once all chunks have been retrieved.

The authenticated request depends on the mechanism. For CURL streaming, some signed secret with a timestamp will probably need to be generated and posted to the file origination site. Then the data can be retrieved with minimal internal processing and dumped directly to disk using stdio buffering. In the case of a Nomad request, the Nomad request packet will be validated, however scheduling chunk batches and re-assembling them could be tricky.

File Backup/Restore

This is much more complicated as we do not have an authenticate web server to request data from. The metadata can be mostly the same, but we need some form of signalling that we will not be fetching the file via the web. This will likely require a client side process to parse each metadata file and locate a file on disk which it is associated with. Then the data would need to be streamed to the destination server with a special endpoint designed for this task. A java app might be the best option here to retain platform neutrality.

Another option would be to use WebDAV for this step. The metadata files would be uploaded first, and then the data files. If a data file corresponded to an existing metadata file, the metadata would be processed; the file stored appropriately, and the metadata file then removed. In this case, photos of scales 4 and 7 would need to be provided in the metadata.

Optionally, this step could also be performed with a filesystem local to the destination server. This would be the highest performance, and a suite of shell-based tools (in the case of Linux) could perform the "client-side" of the task.

The complexity of this task mandates careful planning into how the data is organised and stored and if necessary backed up remotely or transmitted for backup by the source website.

Backward Compatibility

There are some obvious issues with making data available for backup or cloning which existed on the system prior to the existence of restore/sync tools. To keep the tools themselves relatively uncomplicated (to the extent possible given the constraints) backward compatibility may have to be preformed by dedicated plugin or addon.

Nomad - A High Level Overview

Here's a high level description of how Nomad works.

In this example, "Indigo" is going to send a public message from his website at "podunk.edu". "Nickordo" is a recipient on another site ("example.com").

Indigo first posts his message at podunk.edu. podunk.edu looks up who should receive the message and finds Nickordo. Nickordo usually posts from example.com so we add that destination to our list of recipients. We may also add other destinations for nickordo and anybody else that is following Indigo's posts.

In this example we find that we only have one known recipient at one known location.

We send a packet to example.com:

{
  "type":"notify",
  "sender":{
    "guid":"kgVFf_1_SSbyqH-BNWjWuhAvJ2EhQBTUdw-Q1LwwssAntr8KTBgBSzNVzUm9_RwuDpxI6X8me_QQhZMf7RfjdA",
    "guid_sig":"PT9-TApzpm7QtMxC63MjtdK2nUyxNI0tUoWlOYTFGke3kNdtxSzSvDV4uzq_7SSBtlrNnVMAFx2_1FDgyKawmqVtRPmT7QSXrKOL2oPzL8Hu_nnVVTs_0YOLQJJ0GYACOOK-R5874WuXLEept5-KYg0uShifsvhHnxnPIlDM9lWuZ1hSJTrk3NN9Ds6AKpyNRqf3DUdz81-Xvs8I2kj6y5vfFtm-FPKAqu77XP05r74vGaWbqb1r8zpWC7zxXakVVOHHC4plG6rLINjQzvdSFKCQb5R_xtGsPPfvuE24bv4fvN4ZG2ILvb6X4Dly37WW_HXBqBnUs24mngoTxFaPgNmz1nDQNYQu91-ekX4-BNaovjDx4tP379qIG3-NygHTjFoOMDVUvs-pOPi1kfaoMjmYF2mdZAmVYS2nNLWxbeUymkHXF8lT_iVsJSzyaRFJS1Iqn7zbvwH1iUBjD_pB9EmtNmnUraKrCU9eHES27xTwD-yaaH_GHNc1XwXNbhWJaPFAm35U8ki1Le4WbUVRluFx0qwVqlEF3ieGO84PMidrp51FPm83B_oGt80xpvf6P8Ht5WvVpytjMU8UG7-js8hAzWQeYiK05YTXk-78xg0AO6NoNe_RSRk05zYpF6KlA2yQ_My79rZBv9GFt4kUfIxNjd9OiV1wXdidO7Iaq_Q",
    "url":"http:\/\/podunk.edu",
    "url_sig":"T8Bp7j5DHHhQDCFcAHXfuhUfGk2P3inPbImwaXXF1xJd3TGgluoXyyKDx6WDm07x0hqbupoAoZB1qBP3_WfvWiJVAK4N1FD77EOYttUEHZ7L43xy5PCpojJQmkppGbPJc2jnTIc_F1vvGvw5fv8gBWZvPqTdb6LWF6FLrzwesZpi7j2rsioZ3wyUkqb5TDZaNNeWQrIEYXrEnWkRI_qTSOzx0dRTsGO6SpU1fPWuOOYMZG8Nh18nay0kLpxReuHCiCdxjXRVvk5k9rkcMbDBJcBovhiSioPKv_yJxcZVBATw3z3TTE95kGi4wxCEenxwhSpvouwa5b0hT7NS4Ay70QaxoKiLb3ZjhZaUUn4igCyZM0h6fllR5I6J_sAQxiMYD0v5ouIlb0u8YVMni93j3zlqMWdDUZ4WgTI7NNbo8ug9NQDHd92TPmSE1TytPTgya3tsFMzwyq0LZ0b-g-zSXWIES__jKQ7vAtIs9EwlPxqJXEDDniZ2AJ6biXRYgE2Kd6W_nmI7w31igwQTms3ecXe5ENI3ckEPUAq__llNnND7mxp5ZrdXzd5HHU9slXwDShYcW3yDeQLEwAVomTGSFpBrCX8W77n9hF3JClkWaeS4QcZ3xUtsSS81yLrp__ifFfQqx9_Be89WVyIOoF4oydr08EkZ8zwlAsbZLG7eLXY"
  },
  "callback":"\/post",
  "version":1,
  "secret":"1eaa6613699be6ebb2adcefa5379c61a3678aa0df89025470fac871431b70467"
}

This packet says the following:

I'm Indigo and here is proof. I'm posting from podunk.edu and here is proof. I've got a package for you. The tracking number is "1eaa6613....".

Example.com accepts this packet and says "whoa, hold on - I don't know you. I want to prove who you are." So Example.com connects to podunk.edu through a "well-known URL" that we use for this purpose and looks up the "guid" mentioned above. It should return a bunch of information, one item of which is a public key. Example.com uses this key to verify the signatures in the message to verify that indeed there is a person named Indigo at podunk.edu. We only need to do this once. (Note that Indigo can post from any location. All we have to do is prove that it's Indigo and that Indigo can prove that he's posting from another site.)

Then example.com disconnects and flags that there's a message waiting at podunk.edu. Either immediately, or whenever the urge hits (depending on how important Indigo is to anybody on this site), example.com "calls" podunk.edu. It says something like this:

{
  "type":"pickup",
  "url":"http:\/\/example.com",
  "callback_sig":"teE1_fLIqfyeCuZY4iS7sNU8jUlUuqYOYBiHLarkC99I9K-uSr8DAwVW8ZPZRK-uYdxRMuKFb6cumF_Gt9XjecCPBM8HkoXHOi_VselzJkxPwor4ZPtWYWWaFtRfcAm794LrWjdz62zdESTQd2JJIZWbrli1sUhK801BF3n0Ye6-X1MWhy9EUTVlNimOeRipcuD_srMhUcAXOEbLlrugZ8ovy2YBe6YOXkS8jj0RSFjsOduXAoVhQmNpcobSYsDvaQS3e3MvE6-oXE602zGQhuNLr7DIMt9PCdAeQo-ZM-DHlZGCkGk4O2oQFCXFzGPqLUMWDACGJfTfIWGoh_EJqT_SD5b_Yi_Wk9S1lj7vb-lmxe5JuIf7ezWzHoBT8vswnZxPYlidH2i9wapdzij9il_qqcCWWHIp7q_XkY_Zj52Z4r4gdmiqM-8y1c_1SDX7hrJFRwqL_PKFbEvyi5nMWTEzqp55Tay5Woiv19STK_H_8ufFfD9AOkYnk6rIOMsk9dn3a5tAFpDRyRndXkBWAXwiJjiND2zjue7BFu7Ty40THXcfYRh1a5XrAXcaGeYuagg-8J9tAufu9_LY3qGazFg8kRBVMOn4M8DRKSIhKj7z4MnbYL0s09gREojy4jqWO3VkaOjP2jUGzoPuUDLasudE1ehWFq0K_MTQNavgmp8",
  "callback":"http:\/\/example.com\/post",
  "secret":"1eaa6613699be6ebb2adcefa5379c61a3678aa0df89025470fac871431b70467",
  "secret_sig":"O7nB4_UJHBXi28Suwl9LBZF9hI_9KGVTgehnUlWF1oYMNRnBbVHB9lzUfAoalvp3STbU3xJbtD_S58tv6MfV7J5j2V_S1W5ex3dulmDGB8Pt_7Fe5mbEPmjQFcfv3Eg5dUjYIuDl0TDScfrHyImj7RZIWHbwd7wWVoMzzDa_o33klpYmKZCBvObCh55bRrlFkXZs_dRuOiPwkfX0C6_XES4OyOIYl45V30rdhmf-STrf4L9dKYy_axQ12RIwRcKychvVLwlUJn3bn9lgNXRRU_HTne-09OPcJbUOdcD3DkFoKOxMULBNKPHzsCau0ICYug7S0EP6LpCom_mW78s08LyVA1vYeFZjevBCiGecj57yIAQDYi6_rpWJfihYaWHRN0oqtScUR4Bdf0bQbEHxMs4zAtrOAxfyJCbi6U1pfnGgzXzB9ulOYGnVGNTF7Ey4K7FOZIBtk0ILY2JfvBUaVvVs8ttagOOHmhWhnbCvrnOFlkNdlce7zoJCSUJENUOCYmTRfwB_Jno5fAzRnrsYU3_Z-l1mzniU_OmUPz8mPEh7PwhkqAiVlyaM-q15gn7l2lAIDk9kp2X_iCme7v4V0ADN_DbpaI_0-6mPw5HLbKrCsA-sxlSMB4DO4lDCHYkauj0l25sbfroRWB_hax1O4Q0oWyOlVJLUqEC5nuUJCCE"
} 

What this message says is: This is example.com, I have proof, and I'm here to pick up a package. Here's the tracking number, and here's proof that this is the tracking number you presumably sent to example.com.

Good enough. Podunk.edu checks out the story and indeed, it is example.com, and yes, there's a package waiting with that tracking number. Here's the package...

{
  "success":1,
  "pickup":{
    "notify":{
      "type":"notify",
      "sender":{
        "guid":"kgVFf_1_SSbyqH-BNWjWuhAvJ2EhQBTUdw-Q1LwwssAntr8KTBgBSzNVzUm9_RwuDpxI6X8me_QQhZMf7RfjdA",
        "guid_sig":"PT9-TApzpm7QtMxC63MjtdK2nUyxNI0tUoWlOYTFGke3kNdtxSzSvDV4uzq_7SSBtlrNnVMAFx2_1FDgyKawmqVtRPmT7QSXrKOL2oPzL8Hu_nnVVTs_0YOLQJJ0GYACOOK-R5874WuXLEept5-KYg0uShifsvhHnxnPIlDM9lWuZ1hSJTrk3NN9Ds6AKpyNRqf3DUdz81-Xvs8I2kj6y5vfFtm-FPKAqu77XP05r74vGaWbqb1r8zpWC7zxXakVVOHHC4plG6rLINjQzvdSFKCQb5R_xtGsPPfvuE24bv4fvN4ZG2ILvb6X4Dly37WW_HXBqBnUs24mngoTxFaPgNmz1nDQNYQu91-ekX4-BNaovjDx4tP379qIG3-NygHTjFoOMDVUvs-pOPi1kfaoMjmYF2mdZAmVYS2nNLWxbeUymkHXF8lT_iVsJSzyaRFJS1Iqn7zbvwH1iUBjD_pB9EmtNmnUraKrCU9eHES27xTwD-yaaH_GHNc1XwXNbhWJaPFAm35U8ki1Le4WbUVRluFx0qwVqlEF3ieGO84PMidrp51FPm83B_oGt80xpvf6P8Ht5WvVpytjMU8UG7-js8hAzWQeYiK05YTXk-78xg0AO6NoNe_RSRk05zYpF6KlA2yQ_My79rZBv9GFt4kUfIxNjd9OiV1wXdidO7Iaq_Q",
        "url":"http:\/\/z.podunk.edu",
        "url_sig":"T8Bp7j5DHHhQDCFcAHXfuhUfGk2P3inPbImwaXXF1xJd3TGgluoXyyKDx6WDm07x0hqbupoAoZB1qBP3_WfvWiJVAK4N1FD77EOYttUEHZ7L43xy5PCpojJQmkppGbPJc2jnTIc_F1vvGvw5fv8gBWZvPqTdb6LWF6FLrzwesZpi7j2rsioZ3wyUkqb5TDZaNNeWQrIEYXrEnWkRI_qTSOzx0dRTsGO6SpU1fPWuOOYMZG8Nh18nay0kLpxReuHCiCdxjXRVvk5k9rkcMbDBJcBovhiSioPKv_yJxcZVBATw3z3TTE95kGi4wxCEenxwhSpvouwa5b0hT7NS4Ay70QaxoKiLb3ZjhZaUUn4igCyZM0h6fllR5I6J_sAQxiMYD0v5ouIlb0u8YVMni93j3zlqMWdDUZ4WgTI7NNbo8ug9NQDHd92TPmSE1TytPTgya3tsFMzwyq0LZ0b-g-zSXWIES__jKQ7vAtIs9EwlPxqJXEDDniZ2AJ6biXRYgE2Kd6W_nmI7w31igwQTms3ecXe5ENI3ckEPUAq__llNnND7mxp5ZrdXzd5HHU9slXwDShYcW3yDeQLEwAVomTGSFpBrCX8W77n9hF3JClkWaeS4QcZ3xUtsSS81yLrp__ifFfQqx9_Be89WVyIOoF4oydr08EkZ8zwlAsbZLG7eLXY"
      },
      "callback":"\/post",
      "version":1,
      "secret":"1eaa6613699be6ebb2adcefa5379c61a3678aa0df89025470fac871431b70467"
    },
    "message":{
      "message_id":"10b049ce384cbb2da9467319bc98169ab36290b8bbb403aa0c0accd9cb072e76@podunk.edu",
      "message_top":"10b049ce384cbb2da9467319bc98169ab36290b8bbb403aa0c0accd9cb072e76@podunk.edu",
      "message_parent":"10b049ce384cbb2da9467319bc98169ab36290b8bbb403aa0c0accd9cb072e76@podunk.edu",
      "created":"2012-11-20 04:04:16",
      "edited":"2012-11-20 04:04:16",
      "title":"",
      "body":"Hi Nickordo",
      "app":"",
      "verb":"post",
      "object_type":"",
      "target_type":"",
      "permalink":"",
      "location":"",
      "longlat":"",
      "owner":{
        "name":"Indigo",
        "address":"indigo@podunk.edu",
        "url":"http:\/\/podunk.edu",
        "photo":{
          "mimetype":"image\/jpeg",
          "src":"http:\/\/podunk.edu\/photo\/profile\/m\/5"
        },
        "guid":"kgVFf_1_SSbyqH-BNWjWuhAvJ2EhQBTUdw-Q1LwwssAntr8KTBgBSzNVzUm9_RwuDpxI6X8me_QQhZMf7RfjdA",
        "guid_sig":"PT9-TApzpm7QtMxC63MjtdK2nUyxNI0tUoWlOYTFGke3kNdtxSzSvDV4uzq_7SSBtlrNnVMAFx2_1FDgyKawmqVtRPmT7QSXrKOL2oPzL8Hu_nnVVTs_0YOLQJJ0GYACOOK-R5874WuXLEept5-KYg0uShifsvhHnxnPIlDM9lWuZ1hSJTrk3NN9Ds6AKpyNRqf3DUdz81-Xvs8I2kj6y5vfFtm-FPKAqu77XP05r74vGaWbqb1r8zpWC7zxXakVVOHHC4plG6rLINjQzvdSFKCQb5R_xtGsPPfvuE24bv4fvN4ZG2ILvb6X4Dly37WW_HXBqBnUs24mngoTxFaPgNmz1nDQNYQu91-ekX4-BNaovjDx4tP379qIG3-NygHTjFoOMDVUvs-pOPi1kfaoMjmYF2mdZAmVYS2nNLWxbeUymkHXF8lT_iVsJSzyaRFJS1Iqn7zbvwH1iUBjD_pB9EmtNmnUraKrCU9eHES27xTwD-yaaH_GHNc1XwXNbhWJaPFAm35U8ki1Le4WbUVRluFx0qwVqlEF3ieGO84PMidrp51FPm83B_oGt80xpvf6P8Ht5WvVpytjMU8UG7-js8hAzWQeYiK05YTXk-78xg0AO6NoNe_RSRk05zYpF6KlA2yQ_My79rZBv9GFt4kUfIxNjd9OiV1wXdidO7Iaq_Q"
      },
      "author":{
        "name":"Indigo",
        "address":"indigo@podunk.edu",
        "url":"http:\/\/podunk.edu",
        "photo":{
          "mimetype":"image\/jpeg",
          "src":"http:\/\/podunk.edu\/photo\/profile\/m\/5"
        },
        "guid":"kgVFf_1_SSbyqH-BNWjWuhAvJ2EhQBTUdw-Q1LwwssAntr8KTBgBSzNVzUm9_RwuDpxI6X8me_QQhZMf7RfjdA",
        "guid_sig":"PT9-TApzpm7QtMxC63MjtdK2nUyxNI0tUoWlOYTFGke3kNdtxSzSvDV4uzq_7SSBtlrNnVMAFx2_1FDgyKawmqVtRPmT7QSXrKOL2oPzL8Hu_nnVVTs_0YOLQJJ0GYACOOK-R5874WuXLEept5-KYg0uShifsvhHnxnPIlDM9lWuZ1hSJTrk3NN9Ds6AKpyNRqf3DUdz81-Xvs8I2kj6y5vfFtm-FPKAqu77XP05r74vGaWbqb1r8zpWC7zxXakVVOHHC4plG6rLINjQzvdSFKCQb5R_xtGsPPfvuE24bv4fvN4ZG2ILvb6X4Dly37WW_HXBqBnUs24mngoTxFaPgNmz1nDQNYQu91-ekX4-BNaovjDx4tP379qIG3-NygHTjFoOMDVUvs-pOPi1kfaoMjmYF2mdZAmVYS2nNLWxbeUymkHXF8lT_iVsJSzyaRFJS1Iqn7zbvwH1iUBjD_pB9EmtNmnUraKrCU9eHES27xTwD-yaaH_GHNc1XwXNbhWJaPFAm35U8ki1Le4WbUVRluFx0qwVqlEF3ieGO84PMidrp51FPm83B_oGt80xpvf6P8Ht5WvVpytjMU8UG7-js8hAzWQeYiK05YTXk-78xg0AO6NoNe_RSRk05zYpF6KlA2yQ_My79rZBv9GFt4kUfIxNjd9OiV1wXdidO7Iaq_Q"
      }
    }
  }
}

And that's the package (the original message). Example.com converts this into a form suitable for viewing by Nickordo and notifies Nickordo that there's a new message. Podunk.edu might discover that there are other packages waiting for example.com. If this happens it may also send any and all other waiting packages at this time. Each has the original tracking number attached.

Hubzilla FAQ

Is there a way to change the Admin account?

Is there a way to have multiple administrators?

Yes, but it's a bit messy at the moment as it is not yet exposed in the UI. To make an account an administrative account, one needs to add 4096 to the account_roles entry in the account table of the database. Likewise, to remove administrative permissions, one must subtract 4096 from the account roles.

I can log in, but there are no posts or webpages

Most likely, your item table has crashed. Run the MySQL command

repair table item;

Login doesn't work, immediately after login, the page reloads and I'm logged out

Most likely, your session table has crashed. Run the MySQL command

repair table session;

When I switch theme, I sometimes get elements of one theme superimposed on top of the other

a) store/[data]/smarty3 isn't writeable by the webserver. Make it so.

b) You're using Midori, or with certain themes, Konqueror in KHTML mode.

My network tab won't load, it appears to be caused by a photo or video

Your PHP memory limit is too low. Increase the size of the memory_limit directive in your php.ini

Contrary to popular belief, the number of users on a hub doesn't make any difference to the required memory limit, rather, the content of an individuals matrix counts. Streams with lots of photos and video require more memory than streams with lots of text.

I have no communication with anybody

You're listening on port 443, but do not have a valid SSL certificate. Note this applies even if your baseurl is http.

Don't listen on port 443 if you cannot use it. It is strongly recommended to solve this problem by installing a browser valid SSL certificate rather than disabling port 443.

How do I update a non-Git install?

1) Backup .htconfig.php 2) Backup everything in store/ 3) Backup any custom changes in mod/site/ and view/site 3) Delete your existing installation 4) Upload the new version. 5) Upload the new version of themes and addons. 6) Restore everything backed up earlier.

What do I need to do when moving my hub to a different server

1) Git clone on the new server. Repeat the process for any custom themes, and addons. 2) Rsync .htconfig.php 3) Rsync everything in store/ 4) Rsync everything in mod/site/ and view/site (these will only exist if you have custom modules) 5) Dump and restore DB.

How do I reinstall an existing hub on the same server?

1)

git reset --hard HEAD

will reset all files to their upstream defaults. This will not reset any local files that do not also exist upstream. Eg, if you have local changes to mod/channel.php, this will reset them - but will not reset any changes in mod/site/channel.php 2) If you absolutely must reinstall - for example, if you need to upgrade operating system - follow the steps for moving to a different server, but instead of using rsync, backup and restore some other way.

Do not reinstall a hub with a fresh database and fresh .htconfig.php unless as a very last resort. Creating a temporary account and ask for help via a support channel for non-trivial reinstalls is preferable to reinstalling fresh.

How do I set the default homepage for logged out viewers?

Use the custom_home addon available in the main addons repository.

What do the different directory mode settings mean?

// Configure how we communicate with directory servers.
// DIRECTORY_MODE_NORMAL     = directory client, we will find a directory (all of your member's queries will be directed elsewhere)
// DIRECTORY_MODE_SECONDARY  = caching directory or mirror (keeps in sync with realm primary [adds significant cron execution time])
// DIRECTORY_MODE_PRIMARY    = main directory server (you do not want this unless you are operating your own realm. one per realm.)
// DIRECTORY_MODE_STANDALONE = "off the grid" or private directory services (only local site members in directory)

• The default is NORMAL. This off-loads most directory services to a different server. The server used is the config:system/directory_server. This setting MAY be updated by the code to one of the project secondaries if the current server is unreachable. You should either be in control of this other server, or should trust it to use this setting.
• SECONDARY. This allows your local site to act as a directory server without exposing your member's queries to another server. It requires extra processing time during the cron polling, and is not recommended to be run on a shared web host.
• PRIMARY. This allows you to run a completely separate 'Network' of directory servers with your own Realm. By default, all servers are on the RED_GLOBAL realm unless the config:system/directory_realm setting is overridden. Do not use this unless you have your own directory_realm.
• STANDALONE. This is like primary, except it's a 'Network' all on it's own without talking to any other servers. Use this if you have only one server and want to be segregated from the Red#Matrix directory listings.